Social Science Statistics Blog

August 25, 2009

are you making causal inferences?

Do you have a research project where you're trying to make causal inferences from observational data? Do you think matching might be a useful technique? Are you wondering how to get reviewers to stop bothering you?! Would you like some free consulting advice and data analysis help?

We're involved in some methodological research in this area and could use some experience exploring different types of data sets. If you are interested, we would be like to help you with your data analyses and inferences (for a limited number of people and a limited time). Our interactions about your data will remain between us; in particular, we promise not to scoop you, criticize you in print, or use your data for any substantive purposes at all. In fact, for most purposes we don't even need to see your dependent variable. We would be interested in reporting in our research a few aggregated statistics that test methods we are developing, but we would only do that with your permission.

If you're interested, can you send us an email?

Many thanks,

Stefano Iacus (stefano.iacus@unimi.it)
Gary King (king@harvard.edu)

Posted by Gary King at 11:02 AM

March 27, 2009

Automated Text Analysis of Political Scientists

This is a volume of 100 essays by political scientists, each with less than about 1,000 words concerning one novel or insufficiently appreciated idea in some area of the discipline (edited by me, Kay Scholzman, and Norman Nie, and in honor of Sidney Verba). (I might have heard a rumor that if you buy two copies, your next article will be accepted on the first round and you'll get a great new job offer!) In any event, the last blurb above says nice things about the organization of the essays, which of course we appreciate, but I especially like that the essays, except for a little fine tuning, were ordered via automated text analysis (using an algorithm Justin Grimmer and I are working on). The number of possible orderings of 100 essays is enormous of course -- a tad less than the number of elementary particles in the universe squared -- and the idea that a "mere" human being can choose an optimal ordering is absurd. We've become accustomed to understanding that computers can do arithmetic far faster than we, but we also need to start to get use to the fact that (with some help from modern statistics) they can also "read" better than us too.

Posted by Gary King at 8:42 AM

October 21, 2008

Dan Hopkins on the (non-)Bradley/Wilder Effect

Dan Hopkins, an IQSS post-doctoral fellow, is getting a lot of press lately for his paper on the vanishing Bradley effect (aka Wilder effect) -- whereby pre-election polls favor black candidates more than electorates. His results indicate that this effect has vanished and he predicts it will have little or no effect in the upcoming U.S. Presidential election. If you missed all the articles in the mainstream media, see this Science Magazine article. But more interesting is his paper on the subject, "No Wilder Effect, Never a Whitman Effect: When and Why Polls Mislead about Black and Female Candidates", which is easily the most extensive and definitive study of its kind; you can find a copy here.

Posted by Gary King at 10:14 AM

October 15, 2008

Who Voted for the Nazis?

While everyone is thinking about how the U.S. presidential election will turn out, I thought some of you might also be interested in a forthcoming Journal of Economic History article on a venerable electoral question -- why a democratic electorate in Germany chose a party which then ended their democracy. The article is "Ordinary Economic Voting Behavior in the Extraordinary Election of Adolf Hitler," by me, Ori Rosen, Martin Tanner, and Alex Wagner. There's also a good SwissInfo news story about our article.

Here's the abstract: The enormous Nazi voting literature rarely builds on modern statistical or economic research. By adding these approaches, we find that the most widely accepted existing theories of this era cannot distinguish the Weimar elections from almost any others in any country. Via a retrospective voting account, we show that voters most hurt by the depression, and most likely to oppose the government, fall into separate groups with divergent interests. This explains why some turned to the Nazis and others turned away. The consequences of Hitler's election were extraordinary, but the voting behavior that led to it was not.

Posted by Gary King at 10:57 AM

August 21, 2008

Credit for making data available

For those who have collected research data and made it available to others, its nice when people thank you. But it would be nicer to receive formal scholarly citation credit and web visibility for your hard work. The Dataverse Network project is designed to get you that credit and visibility.

The idea is to give you a free "dataverse" (your view of the universe of data) -- which is a virtual archive where you can store, permanently preserve, and distribute your data (or list data from other dataverses) with everyone or only those you approve. Your dataverse is branded as yours, with the look and feel of your web site and on your web site, but since it is served out by an installation of the Dataverse Network at Harvard you needn't install any software or hardware. Some other features include:

• Safe and permanent data storage in preservation format branded as yours.
• No need to translate data when statistical software formats change.
• Can be easily re-branded if you move institutions, but either way will never be lost.
• Formal citation credit for your data, including a globally unique identifier and universal numeric fingerprint.
• Establish an unbreakable link between your data and related published work.
• Easy ways for others to find your data and associated scholarship.
• Share your data with everyone, or those who sign your licensing agreement, or only individuals or groups you approve.
• Allow users to subset, recode, and download your data in any format
• Run many advanced statistical methods via a GUI on-line.

For an example, go to my homepage at http://gking.harvard.edu and click on dataverse. To get your own dataverse, go to the IQSS Dataverse Network, http://dvn.iq.harvard.edu. For more information on our open source Dataverse Network project, see http://TheData.org.

Posted by Gary King at 10:24 AM

February 14, 2008

Open Access to All of Harvard's Articles

Consider this scenerio. I write a paper. I put it on my web site. The site now gets about 5 million hits a year. (Even if most of them are looking for directions to Gary Indiana, that's a fair amount of distribution.) But if I get lucky and the paper is published in the lead journal in some academic field, the journal prints around 15,000 copies and I'm supposed to take it off my web site. In what universe does this make sense?

The Faculty of Arts and Sciences has at Harvard has now taken action to avoid this situation and adopted this policy:

The Faculty of Arts and Sciences of Harvard University is committed to disseminating the fruits of its research and scholarship as widely as possible. In keeping with that commitment, the Faculty adopts the following policy: Each Faculty member grants to the President and Fellows of Harvard College permission to make available his or her scholarly articles and to exercise the copyright in those articles. In legal terms, the permission granted by each Faculty member is a nonexclusive, irrevocable, paid-up, worldwide license to exercise any and all rights under copyright relating to each of his or her scholarly articles, in any medium, and to authorize others to do the same, provided that the articles are not sold for a profit. The policy will apply to all scholarly articles written while the person is a member of the Faculty except for any articles completed before the adoption of this policy and any articles for which the Faculty member entered into an incompatible licensing or assignment agreement before the adoption of this policy. The Dean or the Dean's designate will waive application of the policy for a particular article upon written request by a Faculty member explaining the need.

To assist the University in distributing the articles, each Faculty member will provide an electronic copy of the final version of the article at no charge to the appropriate representative of the Provost's Office in an appropriate format (such as PDF) specified by the Provost's Office. The Provost's Office may make the article available to the public in an open-access repository.

What do you think? Do you think your university could (or should) adopt this? (For more information, see this site.)

Posted by Gary King at 3:28 PM

October 1, 2007

The Changing Evidence Base of Political Science Research

Kay Schlozman and Norman Nie and I are preparing an edited volume in honor of Sidney Verba. The volume is entitled Political Science: What Should We Know? What Should They Know?. Instead of the usual 10 or so chapters representing something other than each contributor's best work, we invited 100 scholars to write about 1,000 words each -- basically one idea (similar to a blog entry) to address to one or both of the questions in the title. I include a draft of mine below. Comments welcome.

The Changing Evidence Base of Political Science Research

I believe the evidence base of political science and the related social sciences are beginning an underappreciated but historic change. As a result, our knowledge of and practical solutions for problems of government and politics will begin to grow at an enormous rate --- if we are ready.

For the last half-century, we have learned about human populations primarily through sample surveys taken every few years, end-of-period government statistics, and in-depth studies of particular places, people, or events. These sources of information have served us well but, as is widely known, are limited: Survey research produces occasional snapshots of random selections of isolated individuals from unknown geographic locations, and the increases in cell phone use and growing levels of nonresponse are crumbling its scientific foundation. Aggregate government statistics are valuable, but in many countries are of dubious validity and are reported only with intentionally limited resolution or after obscuring valuable information. One-off in-depth studies are highly informative but for the most part do not scale, are not representative, and do not measure long-term change.

In the next half-century, these existing data collection mechanisms will surely continue to be used and improved --- such as with inexpensive web surveys, if the problems with their representativeness can be addressed --- but they will be supplemented by the profusion of massive data bases already becoming available in many areas. Some produce extensive or continuous time information on individual political behavior and its causes, such as based on text sources (via automated information extraction from blogs, emails, speeches, government reports, and other web sources), electoral activity (via ballot images, precinct-level results, and individual-level registration, primary participation, and campaign contribution data), commercial activity (through every credit card and real estate transaction and via product RFIDs), geographic location (by carrying cell phones or passing through toll booths with Fastlane or EZPass transponders), health information (through digital medical records, hospital admittances, and accelerometers and other devices being included in cell phones), and others. Parts of the biological sciences are now effectively becoming social sciences, as developments in genomics, proteomics, metabolomics, and brain imaging produce huge numbers of person-level variables. Satellite imagery is increasing in scope, resolution, and availability. The internet is spawning numerous ways for individuals to interact, such as through social networking sites, social bookmarking, comments on blogs, participating in product reviews, and entering virtual worlds, all of which are possibilities for observation and experimentation. (Ensuring privacy and protection of personal information during the analyses to be conducted with this information will require considerable effort, care, and new work in research ethics, but should not be markedly more difficult than the now routine medical research involving experiments on human subjects with drugs and surgical procedures of unknown safety and efficacy.)

The analogue-to-digital transformation of numerous devices people own makes them work better, faster, and less expensively, but also enables each one to produce data in domains not previously accessible via systematic analysis. This includes everything from real-time changes in the web of contacts among people in in society (the bluetooth in your cell phone knows whether other people are nearby!) to records kept of individuals' web clicking, searches, and advertising clickthroughs. Partly as a result of new technology, governmental bureaucracies are improving their record keeping by moving from paper to electronic data bases, many of which are increasingly available to researchers. Some governmental policies are furthering these changes by requiring more data collection, such as the ``No Child Left Behind Act'' in education and via the proliferation of randomized policy experiments. All these changes are being supplemented by the replication movement in academia that encourages or requires social scientists to share data we have created with other researchers.

These data put numerous advances within our reach for the first time. Instead of trying to extract information from a few thousand activists' opinions about politics every two years, in the necessarily artificial conversation initiated by a survey interview, we can use new methods to mine the tens of millions of political opinions expressed daily in published blogs. Instead of studying the effects of context and interactions among people by asking respondents to recall their frequency and nature of social contacts, we now have the ability to obtain a continuous record of all phone calls, emails, text messages, and in-person contacts among a much larger group. In place of dubious or nonexistent governmental statistics to study economic development or population spread in Africa, we can use satellite pictures of human-generated light at night or networks of roads and other infrastructure measured from space during the day. The number, extent, and variety of questions we can address are considerable and increasing fast.

If we can tackle the substantial privacy issues, build more powerful and more widely applicable theories with observable implications in these new forms of data, help create informatics techniques to ensure that the data are accessible and preserved, and develop new statistical methods adapted to the new types of data, political science can make more dramatic progress than ever before. The challenge before us as a profession, before each of us as researchers, and before the broader community of social scientists, is to prepare for the collection and analysis of these new data sources, to unlock the secrets they hold, and to use this new information to better understand and ameliorate the major problems that affect society and the well-being of human populations.

original PDF version

Posted by Gary King at 8:05 AM

September 21, 2007

Great interactive data displays

This is a video worth watching: Hans Rosling: Debunking third-world myths

Posted by Gary King at 12:27 PM

May 12, 2007

Does Multiple Imputation Make Up Data?

This is a common question, commonly misunderstood. It certainly does seem like MI makes up data, since if you look at the 5 or so imputed data sets, the missing values are indeed filled in. But in fact, the point of MI has nothing to do with making up data, and everything to do merely with putting the data in a more convenient format.

The fact is that the vast majority of our statistical techniques require rectangular data sets, and so data that look like swiss cheese make it really hard to do anything sensible with directly. Listwise deletion, where you excise horizontal slices out of the cheese wherever you see holes, discards a lot of cheese! What MI does instead is to fill in the holes in the data using all available information from the rest of the data set (thus moving some information around) and adding uncertainty to these imputations in the form of variation in the values across the different imputed data sets (thus taking back assertions of knowledge from the imputations when it is not predictable from the rest of the data and from duplication of the same information in different places in the data). If done properly, MI merely puts the data in a convenient rectangular format and enables the user (with some simple combining rules) to apply statsitical techniques to data acting as if it were fully observed. MI standard errors then are not too small, which would be the case if data were being made up.

The particular models for imputation can be used incorrectly or inappropriately (and so should be used with priors when additional information is available; see e.g., "What to do About Missing Values in Time Series Cross-Section Data"), but proper usage of MI makes up no information other than that genuinely available.

Posted by Gary King at 4:01 PM

January 11, 2007

Authors Committee Chair Change

I'd like to announce a change today in our Blog Author's Committee Chair from Amy Perfors to Mike Kellermann. Mike was a Michigan undergrad, spent a year in London at the LSE, and is now a 4th year graduate student in the Government Department at Harvard. He studies political parties in advanced industrial democracies and is interested in statistical and formal approaches to the study of politics. He tells me that writing for the blog is a good way to take his mind off of the collapse of Michigan sports teams in the last year (the Pistons, the Tigers, Michigan football, etc.), but I think its only because his quiet and careful demeanor belies the enormous energy he brings to research and methods.

We are fortunate to have Amy continuing to write for the blog at the same time as she continues her Bayesian studies of language, how kids learn, and a variety of other interesting issues in cognitive science.

Posted by Gary King at 10:40 AM

December 14, 2006

Misunderstandings among Experimentalists and Observationalists

We had several discussions a while ago on this blog about balance test fallacies, and an early version of a paper on the subject that Kosuke Imai, Liz Stuart and I wrote. Kosuke, Liz, and I also had a number of interesting discussions with people in several other fields about this topic, and we've found much confusion about the benefits of the key portions of the major research designs. Observationalists seem to have experiment-envy, which is in at least some cases unwarrented, and experimentalists have their own related issues too. To sort these issues out (largely or at least at first for ourselves), we have now written a new paper that tries to clarify these issues and also incorporates the points from the previous paper (material from the previous paper is the last few pages of this one). We'd be very grateful for any comments anyone might have.

"Misunderstandings among Experimentalists and Observationalists: Balance Test Fallacies in Causal Inference" by Kosuke Imai, Gary King, and Elizabeth Stuart.

Abstract

We attempt to clarify, and show how to avoid, several fallacies of causal inference in experimental and observational studies. These fallacies concern hypothesis tests for covariate balance between the treated and control groups, and the consequences of using randomization, blocking before randomization, and matching after treatment assignment to achieve balance. Applied researchers in a wide range of scientific disciplines seem to fall prey to one or more of these fallacies. To clarify these points, we derive a new three-part decomposition of the potential estimation errors in making causal inferences. We then show how this decomposition can help scholars from different experimental and observational research traditions better understand each other's inferential problems and attempted solutions. We illustrate with a discussion of the misleading conclusions researchers produce when using hypothesis tests to check for balance in experiments and observational studies.

Posted by Gary King at 9:26 AM

November 22, 2006

Business Information and Social Science Statistics, Part II

I mentioned in this earlier blog entry an interview I did with DM Review. Here's the sequel.

Posted by Gary King at 2:25 PM

November 11, 2006

Business Information and Social Science Statistics

I thought readers might be interested in an interview I did with DM Review, a widely read publication in the business world, focusing on what they call "business intelligence, analytics, and data warehousing," which is something close to what we social science statistical analysis -- including programs, open source software like R, statistical methods, informatics, etc. They are very interested in what we do, as more sophisticated methods can probably help them a great deal, and the business world certainly has some terrific data sets that would help out what we do. The interview also covers some of the ongoing research at the Institute for Quantitative Social Science.

If you're interested, see Open BI Forum Goes to Harvard ("BI" is jargon for "business intelligence").

Posted by Gary King at 4:18 PM

October 18, 2006

The probability of vanishing posts

There was for a while a post on this blog with comments about an article on deaths in the Iraq war. Despite many good points all parties had in this discussion, it was distracting some folks here from our mission to make the world safe for quantitative methodology. If you're interested in reading more about this subject, we recommend Andrew Gelman's blog post on this subject, which includes many of those who posted and commented here. Thanks for the original post and everyone who commented, and sorry for the confusion.

Posted by Gary King at 4:20 PM

October 11, 2006

data (non)sharing

Jeremy Freese, an RWJ Health Policy Scholar at IQSS this year, sent me this amazing abstract (below) from the front lines of the replication movement, in psychology. On the same topic, but different discipline, don't miss Jeremy's "Reproducibility Standards in Quantitative Social Science: Why Not Sociology?" (find the pdf at his homepage) forthcoming, Sociological Methods and Research, July 2006. (I've written some on this topic too).

"The Poor Availability of Psychological Research Data for Reanalysis" By Wicherts, Jelte M.; Borsboom, Denny; Kats, Judith; Molenaar, Dylan American Psychologist. 61(7), Oct 2006, 726-728.

Abstract

The origin of the present comment lies in a failed attempt to obtain, through e-mailed requests, data reported in 141 empirical articles recently published by the American Psychological Association (APA). Our original aim was to reanalyze these data sets to assess the robustness of the research findings to outliers. We never got that far. In June 2005, we contacted the corresponding author of every article that appeared in the last two 2004 issues of four major APA journals. Because their articles had been published in APA journals, we were certain that all of the authors had signed the APA Certification of Compliance With APA Ethical Principles, which includes the principle on sharing data for reanalysis. Unfortunately, 6 months later, after writing more than 400 e-mails--and sending some corresponding authors detailed descriptions of our study aims, approvals of our ethical committee, signed assurances not to share data with others, and even our full resumes-we ended up with a meager 38 positive reactions and the actual data sets from 64 studies (25.7% of the total number of 249 data sets). This means that 73% of the authors did not share their data.

Posted by Gary King at 7:38 PM

October 6, 2006

Andrew Gelman

Boston Chapter of the American Statistical Association Evening Lecture Series

"Rich state, poor state, red state, blue state: What's the matter with
Connecticut? A demonstration of multilevel modeling"

Andrew Gelman, Columbia University*

IQSS, 1737 Cambridge Street, Room N354

Monday, October 9, 7:30pm

* Andrew will also be on hand at IQSS on the morning of Tuesday, October 10, to answer any stats questions you might have.�

Posted by Gary King at 12:31 PM

August 1, 2006

More on Standards and Statistical Measures of Partisan Gerrymandering

For those interested in more detail about the Texas Redistricting case, and associated Amici brief, that Drew Thomas wrote about a few entries ago, you might be interested in The Future of Partisan Symmetry as a Judicial Test for Partisan Gerrymandering after LULAC v. Perry, by Bernie Grofman and me, forthcoming in the Election Law Journal. An abstract appears below. Comments welcome!

While the Supreme Court in Bandemer v. Davis found partisan gerrymandering to be justiciable, no challenged redistricting plan in the subsequent 20 years has been held unconstitutional on partisan grounds. Then, in Vieth v. Jubilerer, five justices concluded that some standard might be adopted in a future case, if a manageable rule could be found. When gerrymandering next came before the Court, in LULAC v. Perry, we along with our colleagues filed an Amicus Brief (King et al., 2005), proposing that a test be based in part on the partisan symmetry standard. Although the issue was not resolved, our proposal was discussed and positively evaluated in three of the opinions, including the plurality judgment, and for the first time for any proposal the Court gave a clear indication that a future legal test for partisan gerrymandering will likely include partisan symmetry. A majority of Justices now appear to endorse the view that the measurement of partisan symmetry may be used in partisan gerrymandering claims as “a helpful (though certainly not talismanic) tool” (Justice Stevens, joined by Justice Breyer), provided one recognizes that “asymmetry alone is not a reliable measure of unconstitutional partisanship” and possibly that the standard would be applied only after at least one election has been held under the redistricting plan at issue (Justice Kennedy, joined by Justices Souter and Ginsburg). We use this essay to respond to the request of Justices Souter and Ginsburg that “further attention … be devoted to the administrability of such a criterion at all levels of redistricting and its review.” Building on our previous scholarly work, our Amicus Brief, the observations of these five Justices, and a supporting consensus in the academic literature, we offer here a social science perspective on the conceptualization and measurement of partisan gerrymandering and the development of relevant legal rules based on what is effectively the Supreme Court’s open invitation to lower courts to revisit these issues in the light of LULAC v. Perry.

Posted by Gary King at 8:44 PM

July 13, 2006

Job: Preceptor in Statistical Methods and Mathematics

Harvard University invites applications for the full-time position of Preceptor in Statistical Methods and Mathematics beginning September 1, 2006. The preceptor is expected to design and teach one or more introductory courses in mathematical and statistical methods, and related computer support, for undergraduate and graduate students in social and health sciences, education, public policy and related areas. In addition, work in collaboration with faculty to build infrastructure and support statistical and mathematical courses in the Government Department in the design of teaching software, coordination with computer support, creation of laboratory and discussion components, and the development and integration of innovative teaching materials. The position also entails significant management responsibilities, including pedagogical mentoring, advanced programming, sample creation, software testing and documentation, and curriculum research and design. Applicants must have a strong background in statistical methods. Acts as primary liaison between instructors and teaching assistants. For more information, please see this ad.

Posted by Gary King at 9:02 PM

June 30, 2006

The Balance Test Fallacy in Matching Methods for Causal Inference

We've talked a lot on this blog about evaluating the quality of matching solutions when applying these matching for preprocessing, and in some of these discussions I've previewed and referenced arguments from a paper I was working on with Kosuke Imai and Liz Stuart. We have finally finished the paper. For anyone interested, you can get a copy here. The abstract follows. Comments welcome!

Matching methods are widely used to adjust for nonrandom treatment assignment when making causal inferences. In numerous articles across a diverse variety of academic fields that use matching, researchers evaluate the success of the procedure by conducting hypothesis tests, most commonly the t-test for the mean difference of each of the observed covariates between the matched treated and control groups. We demonstrate that these hypothesis tests are fallacious and discuss better alternatives.

Posted by Gary King at 7:47 AM

April 10, 2006

How many combinations have never been chosen at the ice cream shop?

To follow up a previous post of mine, here's another statistics-related lesson to do with your kids. I came up with it at an ice cream shop with my 10-year-old daughter a couple of weeks ago. The point of the lesson is about the power of combinatorics and really, really big numbers. The result is pretty surprising. Here's the recipe:

INGREDIENTS: An ice cream shop, some money for some ice cream, a kid, and a calculator. [I hear 2 objections. To the first: Don't worry, you're probably already carrying a calculator; look closer at your cell phone. The second is: shouldn't we be requiring kids to make the calculations themselves? The fact is that lots of famous mathematicians and statisticians are pretty bad at arithmetic, even though they are obviously spectacularly good at higher level mathematics. Being able to multiply 2-digit numbers in your head is probably useful for something, but understanding the point of the calculation -- why you're doing it, what the inputs are, and what the result of the calculation means -- is far more important.]

DIRECTIONS: Make your order, sit down, and, while you're eating, pose this question to your kid: Suppose the choices on the menu on the wall have never changed since the shop opened. How many choices do you see that have never been chosen even once?

After thinking about weird but fun options like pouring coffee in an ice cream cone, we try it a little more systematically. So we first set out to figure out how many options there are. So I ask, "how many ice cream flavors are there?" My daughter counts them up; it was 20. So how many combinations of one flavor can you have? 20 obviously. How many combinations of two flavors can you have (where for simplicity, we'll count a cone with chocolate on the bottom and vanilla on the top as different from the reverse)? The answer is 20 x 20 or 400. (Its not 40, its 400. Think of a checkerboard with one flavor down the 20 rows and another across the 20 columns and the individual squares as the combination of the two.)

So how many toppings could we have on that ice cream? She went to the counter and counted: 18. And then did 18*400, which she figured out is 7,200. After that we used the calculator and just continued to multiply and multiply as I point out categories on the menu and she counts each up. The total gets big very fast. We got to numbers in the trillions in just a few minutes.

So we find that the total number of options is a really big number. But what does that say about how many options have been tried?

Let's suppose, I say, that it only takes one second for someone to make their choice and receive their order, and that the shop is open 24 hours a day, 7 days a week, all year round. (You could make more realistic assumptions, and teach some good data collection techniques, by watching people get their orders and timing them.) Then we figure out how long it would take for the shop to have been open (under these wildly optimistic assumptions) in order to serve up all the options. To calculate the number of years, all you do is take the number of options, divide by 60 (seconds a minute), 60 (minutes an hour), 24 (hours in a day), and 365 (days a year). In our case, to serve all the options, the shop would have had to be open for around 43,000 years!

So even if the shop had been open for 100 years, it couldn't have served even a tiny fraction of the available options. So how many choices have never been tried at the ice cream shop? Its not just the few that we can cleverly dream up. In fact, almost all of them (over 99 percent of the possibilities) have never been tried!
(At which point my daughter said, "ok, let's get started!")

Actually, if you go to a deli and try this, you can get much larger numbers. For example, if the menu has about 85 items, and each one can be ordered in 10 different ways, the number of possible orders (10 to the 85) is larger than the number of elementary particles in the universe.

Posted by Gary King at 6:00 AM

April 3, 2006

A Unified Theory of Statistical Inference?

If inference is the process of using data we have in order to learn about data we do not have, it seems obvious that there can never be a proof that anyone has arrived at the "correct" theory of inference. After all, the data we have might have nothing to do with the data we don't have. So all the (fairly religious) attempts at unification -- likelihood, Bayes, Bayes with frequentist checks, bootstrapping, etc., etc. -- each contribute a great deal but they are unlikely to constitute The Answer. The best we can hope for is an agreement, or a convention, or a set of practices that are consistent across fields. But getting people to agree on normative principles in this area is not obviously different from getting them to agree on the normative principles of political philosophy (or any other normative principles).

It just doesn't happen, and even if it does it would have merely the status of a compromise rather than the correct answer, the latter being impossible.

Yet, there is a unifying principal that would represent progress in the sense that would advance the field: we will know that something like unification has occurred when we distribute the same data, and the same inferential question, to a range of scholars with different theories of inference, that go by different names, use different conventions, and are implemented with different software, and yet they all produce approximately the same emprical answer.

We are not there yet, and there are some killer examples where the different approaches yield very different conclusions, but there does appear to be some movement in this direction. The basic unifying idea I think is that all theories of inference require some assumptions, but we should never take any theory of inference so seriously that we don't stop to check the veracity of the assumptions. The key is that conditioning on a model does not work, since of course all models are wrong, and some are really bad. What I notice is that most of the time, you can get roughly the same answers using (1) likelihood or Bayesian models with careful goodness of fit checks and adjustments to the model if necessary, (2) various types of robust, semi-parametric, etc. statistical methods, (3) matching for use as preprocessing data that is later analyzed or further adjusted by parametric likelihood or Bayesian methods, (4) Bayesian model averaging, with a large enough class of models to average over, (5) the related "committee methods'', (6) mixture of experts models, and (7) some highly flexible functional forms, like neural network models. Done properly, these will all usually give similar answers.

This is related to Xiao-Li Meng's self-efficiency result: the rule that ``more data are better'' only holds under the right model. Inference can't be completely automated for most quantities, and we typically can't make inferences without some modeling assumptions, but the answer won't be right unless the assumptions are correct, and we can't ever know that the assumptions are right. That means that any approach has to come to terms with the concept that some of the data might not be right for the given model, or the model might be wrong for the observed data. Each of the approaches above has an extra component to try to get around the problem of incorrect models. This isn't a unification of statistical procedure, or a single unified theory of inference, but it may be leading to a unificiation of results of many diverse procedures, as we take the intuition from each area and apply it across them all.

Posted by Gary King at 6:00 AM

March 23, 2006

Control Groups for Breakfast, Revisited

A few months ago, I wrote an entry entitled The Value of Control Groups in Causal Inference (and Breakfast Cereal). It was a report on a fun experiment I did that worked well both in my daughter's kindergarten class and my graduate methods class at Harvard. There were a fair number of comments posted in the blog, and I also received dozens of other notes from parents and school teachers all over the country with many interesting questions and suggestions.

That correspondence covered four main points:

1. Some people suggested a variety of interesting alternative experiments, which is great, but in designing these many forgot that you must always have a control group. That's the main lesson of the experiment: you often learn nothing without some kind of control group, and teaching this to kids (and graduate students!) is quite important.
2. Some people didn't squish the cereal enough and the magnet didn't pick up the pieces. It will attract only when squished very well since the bits of iron are very small.
3. People then asked why the cereal doesn't stick to the magnet without squishing it up. The reason is the same reason a magnet won't pick up a nail driven into a log, but it will pick up the nail if not in the log.
4. Finally, most people asked for other experiments they could run with their kids. For that, which I'm writing up now, please tune in next time!

Posted by Gary King at 6:00 AM

March 14, 2006

So You Want to Do a Survey?

"I'm doing a survey. I've never done this before, taken any classes on survey research, or read any books on the subject, and a friend suggested that I get some advice. Can you help me? I'm going in the field next week."

Someone has asked me versions of this question almost every month since I was a graduate student, and every time I have to convey the bad news: doing survey research right is extremely difficult. The reason the question keeps coming up is that it seems like a such a reasonable question: what could be hard about asking questions and collecting some answers? What could someone do wrong that couldn't be fixed in a quick conversation? Don't we ask questions informally in casual conversation all the time? Why can't we merely write up some questions, get some quick advice from someone who has tried this before, and go do a survey?

Well, it may seem easy, but survey research requires considerable expertise, not any less than heart surgery or flying military aircraft. Survey research should not be done casually if you care about the results. Survey research seems easy because its possible to learn a little without much expertise, whereas doing a little heart surgery with a dinner knife, or grabbing the keys to a B-2 after seeing Top Gun, wouldn't accomplish anything useful.

Survey research is not easy; in fact, its a miracle it works at all. Think about it this way. When was the last time you had a misunderstanding with your spouse, a miscommunication with your parent or child, or your colleague thought you were saying one thing and you meant another? That's right: you've known these people for decades and your questions are still misunderstood. When was the last time your carefully worded, and extensively rewritten article or book was misunderstood? This happens all the time. And yet you think you can walk into someone's home you've never met, or do a cold call on the phone, and in five minutes elicit their inner thoughts without error? Its hard to imagine a more arrogant, unjustified assumption.

So what's a prospective survey researcher to do? Taking a course, reading some books, etc., would be a good start. Our blog has discussed some issues in survey research before, such as in this entry and this one on using anchoring vignette technology to deal with the problem of survey respondents who may interpret survey questions differently from each other and from the investigator. Issues of missing data arise commonly in survey research too. I'm sure we'll discuss lots of other survey-related issues on this blog in the future as well.

A more general facility for information on the subject is the Institute for Quantitative Social Science's Survey Research Program, run by Sunshine Hillygus. This web site has a considerable amount of information on the art and science of questioning people you don't know on topics they may know. If readers are aware of any resources not listed on this site that may be of help survey researchers, please post a comment!

Posted by Gary King at 6:00 AM

February 8, 2006

New Author's Committee Chair

I'd like to announce a change today in our Blog Author's Committee Chair from Jim Greiner to Amy Perfors. Amy was a Stanford undergrad and is now a 3rd year graduate student at MIT. She is interested in using Bayesian technology as models of how humans think, evolutionary linguistics, how humans learn, and a variety of other interesting topics. See her web site for lots more info. In addition to writing some of our most interesting blog entries, I especially recommend this great picture of her winning a line out inrugby!

Jim, the first chair of our author's committee, led this group from a pretty good idea to, in my view and judging from our large and fast growing readership, an enormously successful and informative blog. He will continue on as a member of our Author's Committee, but he's busy this semester running his innovative class in the Law School and Statistics Department, Quantitative Social Science, Law, Expert Witnesses, and Litigation.

Jim Greiner graduated with a B.A. in Government from the University of Virginia and received a J.D. from the University of Michigan Law School in 1995. He clerked for Judge Patrick Higginbotham on the U.S.
Court of Appeals and was a practicing lawyer in the Justice Department and private practice before joining the Harvard Statistics Department.

Posted by Gary King at 6:00 AM

January 21, 2006

Citing and Finding Data

How much slower would scientific progress be if the near universal standards for scholarly citation of articles and books had never been developed. Suppose shortly after publication only some printed works could be reliably found by other scholars; or if researchers were only permitted to read an article if they first committed not to criticize it, or were required to coauthor with the original author any work that built on the original. How many discoveries would never have been made if the titles of books and articles in libraries changed unpredictably, with no link back to the old title; if printed works existed in different libraries under different titles; if researchers routinely redistributed modified versions of other authors' works without changing the title or author listed; or if publishing new editions of books meant that earlier editions were destroyed? How much less would we know about the natural, physical, and social worlds if the references at the back of most articles and books were replaced with casual mentions, in varying, unpredictable, and incomplete formats, of only a few of the works relied on?

These questions are all obviously counterfactuals when it comes to printed matter, but remarkably they are entirely accurate descriptions of our [in]ability to reliably cite, access, and find quantitative data, all of which remain in an entirely primitative state of affairs.

Micah Altman and I have just written a paper on this subject that may be of interest. The title is "A Proposed Standard for the Scholarly Citation of Quantitative Data" and a copy can be found here. The abstract follows. Comments welcome!

An essential aspect of science is a community of scholars cooperating and competing in the pursuit of common goals. A critical component of this community is the common language of and the universal standards for scholarly citation, credit attribution, and the location and retrieval of articles and books. We propose a similar universal standard for citing quantitative data that retains the advantages of print citations, adds other components made possible by, and needed due to, the digital form and systematic nature of quantitative data sets, and is consistent with most existing subfield-specific approaches. Although the digital library field includes numerous creative ideas, we limit ourselves to onl those elements that appear ready for easy practical use by scientists, journal editors, publishers, librarians, and archivists.

Posted by Gary King at 5:50 PM

January 10, 2006

New Professor Joins Government and IQSS

I'm thrilled to announce that Adam Glynn, Ph.D. candidate in the Department of Statistics at the University of Washington, has accepted the offer of the Government Department to be an Assistant Professor here. Adam is a political methodologist and will also be a resident faculty member at the Institute for Quantitative Social Science. His recent work shows how to improve ecological inferences with small, strategically selected samples of individuals. And as it turns out, he can also do the reverse: his work uses ecological inferences from aggregate data to adjust the relationships among the variables in survey data in a manner better than the sometimes current practice of adjusting only the marginals. He has also done work in a variety of other interesting areas. Welcome Adam!

Posted by Gary King at 10:57 PM

October 31, 2005

The Value of Control Groups in Causal Inference (and Breakfast Cereal)

Gary King

A few years ago, I taught the following lesson in my daughter's kindergarden class and my graduate methods class in the same week. It worked pretty well in both. Anyone who has a kid in kindergarten, some good graduate students, or both, might want to try this. It was especially fun for the instructor.

To start, I hold up some nails and ask "does everyone likes to eat nails?" The kindergarten kids scream, "Nooooooo." The graduate students say "No," trying to look cool. I say I'm going to convince them otherwise.

I hand out a little magnet to everyone. I ask the class to figure out what it sticks to and what it doesn't stick to. After a few minutes running around the classroom, the kindergardners figure out that magnets stick to stuff with iron in it, and anything without iron in it doesn't stick. The graduate students sit there looking cool.

From behind the table, I pull out a box of Total Cereal (teaching is just like doing magic tricks, except that you get paid more as a magician). I show them the list of ingredients; "iron, 100 percent" is on the list. I ask by a show of hands whether this is the same iron as in the nails. 3 of 23 kindergarten kids say "yes"; 5 of 44 Harvard graduate students say "yes" (almost the same percent in both classes!).

I show the students that the box is sealed (and I have nothing up my sleeves), Then, I open the box, spill some cereal on a cutting board, and smash it up into tiny pieces with a rolling pin. I take the pile of cereal around the room and let the kids put their magnet next to it and see whether the cereal sticks to the magnet. To everyone's amazement, it sticks!

Then I ask, are we now convinced that the iron in the nails is the same iron as in the cereal? All the kids in kindergarten and all the graduate students say "yes."

I respond by saying "but how do you know the cereal stuck to the magnet because it had iron in it? Maybe it was just sticky, like gum or tape." Now that I finally have their attention (not a minor matter with kindergartners), I get to explain to them what a control group is. And from behind the table, I pull out a box of Rice Krispies (which are made of nothing). We examine the side of the box to verify the lack of (much) iron, and then I smash up the Rice Krispies, and let them see if their magnet sticks. It doesn't stick!

Everyone gets to take home a cool fact (they love to eat the stuff in nails), I get to convey the point of the lesson in a way they won't forget (the central role of control groups in causal inference), and everyone gets a free magnet.

Posted by Gary King at 2:18 AM

October 27, 2005

Don't Use Hypothesis Tests for Balance

Gary King

Jens' last two blog posts constitute an excellent statement of where the literature on matching is, but I think almost all of the literature has this point wrong. Hypothesis tests for checking balance in matching are in fact (1) unhelpful at best and (2) usually harmful.

Suppose you had a control group and a treatment group that are identical (exactly matched) except for one person, or except for a bunch of people in one very minor way. Suppose hypothesis tests indicate no difference between the groups, and so you'd be in the situation of reporting balance was great and no further adjustment was needed. (We might think of this as a real experiment where the outcome variable hasn't been collected but is expensive to do so.) If you were given the chance of dropping the one or few people that caused the two groups to differ and replacing them with others that exactly matched, would you do so? Since the dimension on which the inexact match or matches occurred might be the one that has a huge effect on your outcome variable, the bias due to not switching could be huge. So you'd undoubtedly make the switch, despite the fact that the hypothesis test indicated that there was no problem. Hence (1) the tests are unhelpful: passing the test does not necessarily protect one from bias more than failing the test.

Now suppose you have data that don't match very well by all hypothesis tests and you randomly (rather than systematically to improve matching) drop observations, in a bad application of matching. what will happen? Your t-tests or ks-tests or any other hypothesis tests will lose power and so will indicate that balance is getting better and better. Yet, bias is not changing at all, and efficency is dropping fast. The tests are telling you to discard data! Hence (2) hypothesis tests to evaluate balance are harmful, quite seriously so.

The fact is that there is no superpopulation to which we need to infer features of the explanatory variables; all analysis models we regularly use after matching are conditional on X. Balance should be assessed on the observed data, and not be the subject of inference or hypothesis tests.

This message rehearses an argument in a to-be-revised version of our matching paper by Ho, Imai, King, and Stuart that we hope to be finished with and post in a couple of weeks.

Posted by Gary King at 4:40 AM

October 25, 2005

New IR Data Set with 10 Million Dyadic Events

I thought you might be interested in a newly updated dataset of almost 10 million individually coded international events (1990-2004). Each event is summarized in the data as "Actor A does something to Actor B", with Actors A and B coded for about 450 countries (and other actors) and "does something to" coded in an ontology of about 200 types of actions. The data are coded by a computer "reading" millions of Reuters news reports. Will Lowe and I wrote an article* that evaluated the software system (produced by VRA) that performs this task and found that for the numbers of events it was possible to convince humans (trained Harvard undergraduates) to coded by hand, the machine did as well as the humans. However, in part since there is only so much pizza you can feed undergraduates, the machine clearly dominates for larger numbers of events. We previously released a dataset with 3.5 million events; this one is bigger, more accurate (since the software has been improved), and covers a longer time period.

Most international relations data are limited to analyses aggregated to the year or month. Yet, as we say in the article, when the Palestinians launch a mortar attack into Israel, the Israeli army does not wait until the end of the calendar year to react. We think there is much to be learned about international relations from data like these. For the data, documentation, and our article, see this site.

Gary

*Gary King and Will Lowe. 2003. "An Automated Information Extraction Tool For International Conflict Data with Performance as Good as Human Coders: A Rare Events Evaluation Design" International Organization, 57, 3 (July, 2003): Pp. 617-642.

Posted by Gary King at 5:31 PM

September 20, 2005

New IQSS Blog on Political Behavior

We'd like to welcome the new Political Behavior Blog at the Institute run by Prof. Barry Burden and his team of graduate students from Harvard and MIT. They've only just started but they have some interesting material on the way. If you're interested in political behavior, we encourage you to check it out. See http://iq.harvard.edu/blog/pb.

Posted by Gary King at 10:56 PM

September 13, 2005

new URL

By popular demand, we've shortened the URL for this blog. The old one still works, but the URL is now: http://iq.harvard.edu/blog/sss/

Posted by Gary King at 1:21 PM

September 12, 2005

Harvard University Faculty Position in Quantitative Political Methodology, Tenure-Track Level

The Department of Government invites applications for a position in quantitative political methodology at the rank of Assistant or untenured Associate Professor to begin July 1, 2006.   Candidates should expect to have completed the requirements for the Ph.D. prior to appointment. Teaching duties will include offering courses at undergraduate and graduate levels. Candidates are expected to demonstrate a promise of excellence both in research and teaching in political methodology. Harvard is an Affirmative Action/Equal Opportunity Employer; applications from women and minority candidates are strongly encouraged. We will begin to review applications on Monday, October 10, 2005 and will continue until the position is filled. Send application, including cv, transcripts, at least 3 letters of recommendation, samples of written material, teaching evaluation materials if available, and a one-page summary of a proposed job talk to: Faculty Recruitment Committee, Political Methodology, Government Department, Harvard University, 1737 Cambridge Street, Cambridge, MA 02138.

Posted by Gary King at 3:59 PM

September 8, 2005

State Failure

This article in World Politics on forecasting state failure that Langche Zeng (who by the way is moving this week from GW to UCSD) and I wrote a few years ago seems relevant to what is presently happening in New Orleans. Here are the opening sentences of the article: "`State failure' refers to the complete or partial collapse of state authority, such as occurred in Somalia and Bosnia. Failed states have little political authority or ability to impose the rule of law [on its citizens]." We normally associate state failure with foreign countries you would not want to visit, but with a third of the New Orleans police force not showing up for work, with the two-thirds that remained barricaded in their homes or police stations, with corpses strewn around the streets from the hurricane and some murders, and where a policeman today "joked that if you wanted to kill someone here, this was a good time" (see today's NY Times Article), it is hard to see how New Orleans this past week was anything but the definition of state failure.

Our article was about some methodological errors we found in the U.S. State Failure Task Force's forecasts and methods of forecasting. They had selected data via a case-control design (i.e., selecting on their dependent variable all examples of state failure and a random sample of nonfailures), which can save an enormous amount of work in data collection, but it is only valid if you properly correct. The Task Force didn't correct and so, for example, their forecast for Brazil failing was reported at 0.72 but their model, correctly interpreted, indicated that it was only 0.11; their reported forecast for Somalia failing was 0.45, but the model actually indicated that it was only 0.04. We also improved their methods and thus forecasting success over their corrected models via neural network methods and some other approaches. They also collected one of the best data sets on the subject, which you might want to use.

The charter of the U.S. State Failure Task Force prohibits it from discussing state failure in the U.S. or making forecasts of U.S. state failure, but by their definitions, there is little doubt that for a time anyway all relevant governmental authorities in the U.S. suffered a "complete or partial collapse of state authority" and so the U.S. would seem to fit that definition. I haven't checked, but I doubt their model or our's had any ability forecast these events.

Posted by Gary King at 8:59 AM

September 6, 2005

Welcome

Welcome to the Social Science Statistics Blog, hosted by the Institute for Quantitative Social Science at Harvard University. We are starting this blog today in order to make public some of the hallway conversations about social science statistical methods and analysis that are regular features at the Institute and related research groups. Perhaps you may have also found that while formally published research is emphasizing one topic or approach, conversations with scholars at conferences reveal a strong trend about to proceed in a new direction. We similarly find correlated trends in the work-in-progress of many of Harvard's methodologists and visitors' informal speculations and plans revealed while making their rounds at our seminars. We find familiarity with these trends to be valuable for our own research, and so we hope to record some of this information here for ourselves, our students, and anyone else who may wish to listen in.

This blog may be especially useful at Harvard given the high level of decentralization here -- referred to by Harvard insiders as "every tub [or School] on its own bottom" although, since this decentralization often goes right down to the individual faculty member, I sometimes think a better phrase might be "everyone with a bottom has their own tub." To prevent this formal structure from having negative intellectual consequences, faculty here often invent structures to span our formal structures. This blog is one of those structures. Another is a weekly Research Workshop on Applied Statistics we started a few years ago, billed as "a tour of Harvard's statistical innovations and applications with weekly stops in different disciplines". Every week during the academic year, a differing subset of the almost 300 faculty and students from across the university who have signed themselves up for our mailing list appear at the Institute for a talk on some aspect of social science methods or their application. Most of us find this regular exchange with such a diverse group of scholars to be highly productive, although we sometimes have to figure out how to translate the jargon describing the same statistical models from one discipline to another (most are familiar with either "Malmquist Bias" in Astronomy or "Selection Bias" in Economics but rarely both, despite the fact that they are almost identical mathematically.)

Although most of our blog posts will involve other subjects, one post each week during the academic year will include summaries of (and when available links to) papers presented at our weekly seminar, along with a sense of the discussion that takes place afterwards. Some of the other topics we plan will include posts on trends in methodological thought, questions and comments, paper and conference announcements, applied problems needing methodological solutions, methodological techniques seeking applied problems, and whatever else may be of interest and occurs to someone around here. Comments on posts are welcome from others too.

The main responsibility for the daily posts on this blog has been taken by an extremely talented group of gradaute students representing six different academic disciplines. Our authoring team is chaired by Jim Greiner, who has a law degree, practical experience with the Justice Department and a law firm in Washington D.C., and is now a Ph.D. candidate in Harvard's Statistics Department. Members of our committee include Sebastian Bauhoff, in the Economics track of the Health Policy Ph.D. Program; Felix Elwert, a Ph.D. candidate in the Department of Sociology and an A.M. candidate in the Department of Statistics; John Friedman, a Ph.D student in the Economics department; Jens Hainmueller and Mike Kellermann, graduate students in the Department of Government; Amy Perfors, a graduate student in the Brain and Cognitive Sciences department at MIT; Andrew (Drew) C. Thomas, who after getting a B.A. in physics from MIT has joined the Department of Statistics Ph.D. program; and Jong-Sung You, a Ph. D. Candidate in Public Policy at the Kennedy School and Doctoral Fellow of Inequality and Social Policy Program. Please read more about our team here.

Posted by Gary King at 12:38 PM