Complexity and Social Networks Blog

I enjoyed your presentation quite a bit, and thanks for linking to my post on the Social Science Stats blog.

I can't speak to the best frameworks for doing computation with large datasets, but I have some experience with collecting text off the web and also with storage. For collecting data off of the web, generally people use a scripting language like Perl, Python, or Ruby (my favorite), each of which has libraries for 1) navigating websites programmatically and 2) parsing the html content. For data storage, I use an open-source database system called MySQL or its lighter-weight cousin sqlite.

Ironically these are the same tools used to create and maintain many websites from which you might collect data. It's always better if you can just get the database files directly rather than doing a bunch of clever scripting to replicate them from scratch, but this isn't always possible.

Hello David et al. I'm currently dealing with a location tracking dataset obtained from a Ultra Wide Band based system which was deployed in an office building and used by 51 people for 6 weeks. I've got thousands of xyz observations (with 4 updates per second)and I'm manipulating it using MATLAB to query the multiple excel spreadsheets I've got, extract some meaningful info and mapping it with the help of GIS software MapInfo. By the way I'm trying to measure the duration and location of interactions in the office.

I would break up the need for computational tools in data analysis into four different types of tasks: (1) data collection, which at this point means data scraping from the web and more or less structured text files; (2) data manipulation, which means transforming the data into a data structure in your favorite computational system in such a way that you can "compute" e.g. execute abstract operations on it); (3) data visualization, which is summarizing data in plots, graphs etc to get a qualitative impression of your data; and (4) data analysis proper, which means applying mathematical and statistical tools to your data.

Programming languages and software packages out there cover different subsets of these tasks, but it seems obvious to meet that it is a good thing if all of these can be performed within a single environment where one task flows naturally into the other. Integrating these four types of tasks is what lies at the heart of the computable data initiative that I helped develop in Mathematica. This is something that could potentially be done in other programming languages and environments (such as Python and R). But I can definitely say that a huge amount of work goes into making all these pieces fit together :-)