Complexity and Social Networks Blog

The logistical advantage is that as the NSA finds out about potentially risky individuals, they can avoid the hurdles of making requests of the telecomms—they could just instantly access the information as they needed it.

Would such a massive data set be useful? Probably. Certainly, the locational information would be very helpful—one would be able to evaluate the physical proximity of people. Further, some of the patterns one would look for would involve the locations of individuals making and receiving calls—a set of calls to different numbers to Washington, DC, from a high risk source might be indicative of a potential event there.

One could also refine the techniques to identify members of a loosely connected set of people by testing them on known sets of people. There has been a lot of work recently on identifying groups of people from network data, e.g., by Ken Frank, Mark Newman, and Bernardo Huberman. I’m not sure how their algorithms would scale up to such a massive data set. I suspect that you could produce some algorithm that could do something similar (if not as well) for a dataset of hundreds of millions, although perhaps I am wrong.

This problem is significantly different, since you would be starting with somewhat more information—e.g., that a handful of people belong to a particular group—and not really want to produce a list of all groups in the data set. Further, you would have more information than just who talks with whom, but when they talked, and through what medium.

So it would be necessary to produce a new, and much more sophisticated, algorithm, testing on a variety of groups where you could validate the results. For example, one could test it on social network analysts—start with a handful of people who you know do social network analysis, and produce an algorithm that does a reasonable job of finding other social network analysts, adjusting the parameters of the algorithm to fit. Repeat this with a variety of groups, until you produced a reasonably robust algorithm.

Hard to say how effective this would be without doing it. And whether any groups that you could validate would have communication patterns like terrorists seems rather unlikely. Of course, you might be willing to settle for lots of false positives if you are looking for a terrorist—e.g., 100 or 1000 false positives to find one true positive.

Of course, such data would have a lot of potential for "collateral" usage, which I will turn to next.

See web pages of Ken Frank, Mark Newman, and Bernardo Huberman.