Complexity and Social Networks Blog

« February 2006 | Main | April 2006 »

28 March 2006

Open source intelligence?

Excerpts from an interesting story in today's NYT, where Iraqi government documents are being placed on the web. What is most illuminating, really, is the discussion of the motivation of the bloggers, etc, who are examining the documents. The questions, in an exercise like this, are (1) why would anyone put substantial effort into something like this? and (2) do meritorious contributions "float to the top?

Re the first, it seems clear in this case, consistent with open source projects, that key motivations are the possibility of "making a difference" and of getting attention (e.g., getting on the front page of the NYT). Re the second, I suspect what will happen, in contrast to open source software projects (which have a certain degree of central management to guarantee continued coherence), is that multiple truths will emerge, where what is ideologically meritorious will float to the top in different communities.

_____________________________________________________________

March 28, 2006
Iraqi Documents Are Put on Web, and Search Is On
By SCOTT SHANE

WASHINGTON, March 27 — American intelligence agencies and presidential commissions long ago concluded that Saddam Hussein had no unconventional weapons and no substantive ties to Al Qaeda before the 2003 invasion.

But now, an unusual experiment in public access is giving anyone with a computer a chance to play intelligence analyst and second-guess the government.

Under pressure from Congressional Republicans, the director of national intelligence has begun a yearlong process of posting on the Web 48,000 boxes of Arabic-language Iraqi documents captured by American troops.

Less than two weeks into the project, and with only 600 out of possibly a million documents and video and audio files posted, some conservative bloggers are already asserting that the material undermines the official view.

On his blog last week, Ray Robison, a former Army officer from Alabama, quoted a document reporting a supposed scheme to put anthrax into American leaflets dropped in Iraq and declared: "Saddam's W.M.D. and terrorist connections all proven in one document!!!"

Not so, American intelligence officials say. "Our view is there's nothing in here that changes what we know today," said a senior intelligence official, who would discuss the program only on condition of anonymity because the director of national intelligence, John D. Negroponte, directed his staff to avoid public debates over the documents. "There is no smoking gun on W.M.D., Al Qaeda, those kinds of issues."

All the documents, which are available on fmso.leavenworth.army.mil/products-docex.htm, have received at least a quick review by Arabic linguists and do not alter the government's official stance, officials say. On some tapes already released, in fact, Mr. Hussein expressed frustration that he did not have unconventional weapons.
...

"As an historian, I'm glad to have the material out there," said John Prados, who has written books on national security, including one that accuses the administration of distorting prewar intelligence. He said the records were likely to shed new light on the Iraqi dictatorship. Some of the documents, also included in a new study by the United States military, already have caused a stir by suggesting that Russian officials passed American war plans to Mr. Hussein's government as the invasion began.

But Mr. Prados said the document release "can't be divorced from the political context."

"The administration is under fire for going to war when there was no threat — so the idea here must be to say there was a threat," he said.

That is already the assertion of a growing crowd of bloggers and translators, almost exclusively on the right. So far they have highlighted documents that refer to a meeting between Osama bin Laden and an Iraqi intelligence officer in Sudan in 1995; a plan to train Arab militants as suicide bombers; and a 1997 document discussing the use of "special ammunition," chemical weapons, against the Kurds.

...

... some intelligence analysts are horrified at exactly the idea that excites Mr. Hoekstra and the bloggers: that anyone will now be able to interpret the documents.

"There's no quality control," said Michael Scheuer, a former Central Intelligence Agency specialist on terrorism. "You'll have guys out there with a smattering of Arabic drawing all kinds of crazy conclusions. Rush Limbaugh will cherry-pick from the right, and Al Franken will cherry-pick from the left."

Conservative publications have pushed for months to have the documents made public. In November, Mr. Hoekstra and Senator Pat Roberts of Kansas, chairman of the Senate Intelligence Committee, asked Mr. Negroponte to post the material.

When that request stalled, Mr. Hoekstra introduced a bill on March 3 that would have forced the posting. Mr. Negroponte began the release two weeks later.

Under the program, documents are withheld only if they include information like the names of Iraqis raped by the secret police, instructions for using explosives, intelligence sources or "diplomatically sensitive" material.

In addition, the intelligence official said, known forgeries are not posted. He said the database included "a fair amount of forgeries," sold by Iraqi hustlers or concocted by Iraqis opposed to Mr. Hussein.

In previous Internet projects, volunteers have tested software, scanned chemical compounds for useful drugs and even searched radiotelescope data for signals from extraterrestrial life.

The same volunteer spirit, though with a distinct political twist, motivates the Arabic speakers who are posting English versions of the Iraqi documents.

"I'm trying to pick up documents that shed light on the political debate," said Joseph G. Shahda, 34, a Lebanese-born engineer who lives in a Boston suburb and is spending hours every evening on translations for the conservative Free Republic site. "I think we prematurely concluded there was no W.M.D. and no ties to Al Qaeda."

Mr. Shahda said he was proud he could help make the documents public. "I live in this great country, and it's a time of war," he said. "This is the least I can do."

Posted by David Lazer at 11:42 PM

23 March 2006

The billion dollar question

Last week, an article in the New York Times reported on the newest developments in a decade-long struggle to modernize the F.B.I. computer system (“Cost Concerns for F.B.I. Computer Overhaul", March 14, 2006). Citing a Justice Department report, it says that the overhaul will likely cost “another half-billion dollars to complete". The same amount has already been spent: After the report by the 9/11 commission revealed that the antiquated computer system might have played a part in the intelligence gaps before 9/11, the F.B.I. reacted by devoting $535 million to its Trilogy Program, a network “designed to provide all FBI offices with better organization, access and analysis of information" (F.B.I. press release). So far, the results have been less than satisfactory: Its core component, a case management system (known as Virtual Case File system), collapsed under technical difficulties and was abandoned by the F.B.I. after it had spent $170 million on it. And a few days ago, an audit by the GAO revealed that the F.B.I. and its contractors spent more than $17 million on “questionable payments" (Washington Post, March 18, 2006). Now, the F.B.I. plans to spend $425 million on a new case management system, partly with the same contractors, named Sentinel.

What puzzled me was a statement by the inspector general’s office of the Justice Department quoted in the Times saying that they were unsure whether the new system, “even if successful, would allow the bureau to share information adequately with other intelligence and law enforcement agencies". The lack of information sharing was one of the main issues pointed out by the 9/11 commission, and yet, after investing a billion dollars, information sharing is not built into the system. It seems to me that the project would benefit enormously from the insights organizational researchers have into information networks. For example, my research shows that large IT projects that start off with an exploration of informational needs are more successful in the “exploitation", or implementation phase. Rather than focusing solely on network technology, the F.B.I. should devote some resources to finding out who needs to talk to whom. Have researchers conduct interviews, run focus groups, maybe even do an ethnography to identify the information network, then build the computer network to support it.

Posted by Maria Binz-Scharf at 8:28 PM

21 March 2006

Social Networks and the Business World

Social Network Theory and its principles are applied by more and more companies in a way that some of us might not be aware of yet. So what we buy, how we rate products/services, post in forums, pictures we upload or present of ourselves on the web is significantly influencing other, likeminded individuals. In return we are influenced by the network cluster we belong to for a specific habit and the like. Collaborative filtering is a key component of using social networks for different purposes. Further information can be found here. Below you will find a list of various industry and application examples:

Social Networking plattforms
There are the obvious social networking online plattforms. Among them are the open business and personal contact manegement oriented like Tribe.net , openbc, friendster or the inivitation only communities like asmallworld. Either planned or already implemented users can take advantage of added services (search functionality, messaging) by paying a monthly fee 10< USD. Furthermore, there are the rather dating/partner match making plattforms like match or eharmony.

Retail/eCommerce
Most of today's ecommerce sites use collaborative filtering to improve sales, cross-,up- and downselling. A prominent example are Amazon's recommendations based on various user behaviours on their website.

Music/Radio
Tapping into our musical tastes Last FM, Genielab or Pandora present us with streaming music. Here the main business model lies in linking to the respective ecommerce sites like Apple's iTunes.

Books
The same applies to the area of what we might want to read next which also serves ecommerce purposes.

Movies and more
MovieLens is a free service provided by GroupLens Research at the University of Minnesota. Whether, you want to book a hotel, whole vacation there are numerous examples of collaborative filtering apps on websites.

Pictures
The most prominent example for sharing, managing and searching for pictures is Flickr or myspace. The latter gaining revenues from online-ads.

Search engines
As I have elaborated in an earlier entry on google bombs the network structure (ties) play an important role in search engine algorithms.

Knowledge Base and OpenSource
The online encyclopedia Wikipedia builds on the power of decentralized, voluntary collaboration building an enourmous depository of multi-language information. Whether it was the development of Linux, Mozilla/Firefox or MySQL all rely on and consist of social networks. Further examples of openSource projects can be found at Sourceforge.

SNA Consulting
As we can see the character and concepts of networks is mainly utilized for recommendations. Actual applications of SNA is done by a few companies and consultants like Rob Cross, IBM, Orgnet or Visiblepath. These companies try to uncover the informal networks within organisations to improve knowledge sharing, initiate change or bridging silos.

Finally, you can always follow latest trends in social network analysis at PNG's subpage on SNA by Ines Mergel.

Posted by Alexander Schellong at 11:15 AM

15 March 2006

The Social Affordances of Email in Japan and America

During a recent presentation at the University of Tokyo I discussed the social affordances of email. I defined social affordances as the social opportunities and constraints provided by technology. (If you want to read more about this topic, see my co-authored paper : "The Social Affordances of the Internet for Networked Individualism," in Journal of Computer Mediated Communication, 8, 3.) After I listed a number of email’s social affordances, one of the audience members pointed out that those affordances only apply to PC based email. By contrast, there exists a substantially different set of affordances for mobile phone based email. Given that my research is only about the use of email in America, my lack of attention to mobile phone email was intentional. There are not enough Americans using this technology for it to be relevant to my current research. Nevertheless, this comment got me thinking about the difficultly of making cross-national generalizations about the social uses of particular technologies. For example, even though the use of PC email is almost as common in Japan as it is in America, the wide-spread use of mobile phone email in Japan may change how the Japanese use PC email.

My recent co-authored report for the Pew Internet & American Life Project shows that American’s use PC email for contacting both strong and weak(er) ties. By contrast, in another paper I find that the Japanese tend to use mobile phone email for contacting their strong ties, and PC email for contacting their weak ties (the paper published in Ito et. al 2005). This suggests that the existence of mobile phone email in Japan alters the use of PC email, even though PC email has the same set of affordances in both countries. I’m now thinking about how the unique combinations of communication technologies available to people in different countries effects their use of PC based email. Any thoughts or references regarding this issue would be welcome.

Posted by Jeff Boase at 4:17 PM

14 March 2006

NYT magazine article: Can Network Theory Thwart Terrorsts?

There was a piece in the NYT Magazine on Sunday on the surveillance program, titled “Can Network Theory Thwart Terrorists?� by Patrick Radden Keefe

It offers some basic ideas in applying network analysis to detecting terrorists, although it contains, I believe, a couple of mistakes. First, it states that “Network academics caution that the field is still in its infancy and should not be regarded as a panacea.� Not a panacea, certainly true; but in its infancy? Certainly not—the modern field can trace its roots to the 1930s with the work of Moreno, Newcomb, and others. However, this is really just a quibble, in that the analysis of very large full network data (e.g., see Nathan Eagle’s posting on his work on the phone data of an entire country) is in its infancy, due to the lack of tools and data until recently.

A more serious apparent error is in the following statement:

“The use of such network-based analysis may explain the administration’s decision, shortly after 9/11, to circumvent the Foreign Surveillance Court. The court grants warrants on a case-by-case basis, authorizing comprehensive surveillance of specific individuals. The NSA program, which enjoys backdoor access to America’s major communication switches, appears to do just the opposite: the surveillance is typically much less intrusive than what a FISA warrant would permit, but it involves vast numbers of people.
In some ways, this is much less alarming than old-fashioned wiretapping. A computer that monitors the metadata of your phone calls and e-mail to see if you talk to terrorists will learn less about you than government agent listening in to the words you speak.�

From what we have seen in the news reporting (excerpted in previous postings), however, the NSA program involves more than analysis of metadata. It also involves recording and content analysis through computer algorithms of the conversations of large numbers of individuals, with snippets selected out for human analysis. Indeed, I believe this was the whole basis of the controversy—the stories on the metadata came after the wiretapping story. While I wouldn’t say that metadata pose a non-intrusion, this is a much bigger deal. Indeed, as I believe I have said in previous postings, I would consider a digital recording of my conversation, content coded, to be a greater intrusion on my privacy than having two FBI agents listen to my conversations, with paper transcripts going into a file somewhere. Those bits do not decay like human memories, or easily lost, like files stored in the bowels of an enormous bureaucracy.

Posted by David Lazer at 9:21 PM

8 March 2006

Network Visualization Tools

I find that when I’m inundated with network data, the best way to get my head around it is through visualization. The human eye seems to be able to identify the important structure and topological dynamics much easier than an algorithm. Over the years I’ve spent most of my time using a Window/Linux application called Pajek. I use a Matlab script to turn an adjacency matrix into files that Pajek can interpret as a network. It supports different shapes, colors, and edges - and even can visualize (more or less) dynamic networks.

One of my Pajek networks from here.

However, things start to break down when the networks go beyond a few hundred nodes. There are several packages for large-scale network visualization - however most come with serious limitations. Walrus creates beautiful networks, but unfortunately they need to be spanning trees.

A 500,000 node Walrus network

There are plenty of other network analysis tools out there - but it’d be great to hear people’s experiences actually using them on real data...

Posted by Nathan Eagle at 5:03 AM

6 March 2006

Brinton Milward on "Dark Networks as Organizational Problems"

H. Brinton Milward, University of Arizona

"Dark Networks as Organizational Problems" (PDF)

This talk attempts to integrate the study of illegal networks into the study of organizations. The talk is a work in progress focused on two dark networks - Al Qaeda and Cocaine Trafficking - faced with hostile governments seeking to control them. In addition public policy issues related to governance and control will be discussed.

For an earlier version of the paper.

PLEASE POST REACTIONS TO THE TALK HERE.

Posted by David Lazer at 11:59 AM