Social Science Statistics Blog

February 8, 2010

Rios in the New York Times

Viri Rios has a great op-ed in the New York Times about mathematical social science and Mexican drug politics.

Posted by Richard Nielsen at 2:46 PM | Comments (0)

February 2, 2010

Natural experiments

The other week, I read Jared Diamond's Guns, Germs, and Steel which managed to get me a little worked up about a pet peeve of mine: the term "natural experiment." Just when I had gotten calmed down, the Polmeth list serve alerted me to an entire issue of Political Analysis devoted to natural experiments. Arghhh...

Don't get me wrong -- in my own research I try to use observational data to make causal claims that are probably far more dubious than anything in the special issue of Political Analysis. I'm highly impressed by the research and I'm even more supportive of social scientists who are looking for "natural experiments" in political science. I just wish we could call them something else because I'm skeptical that they are really experimental.

The lead article of the PA special issue urges scholars "to use the language of experimental design in explicating their own research designs and in evaluating those of other scholars." I'm on board with using the language of experiments, but I've also seen more than a few recent papers framed as "natural experiments" that are really just observational studies with no particular claim to special status. The spread of experimental language into observational studies may have downsides as well as benefits.

Until recently, I basically assumed that when people said they had a natural experiment, what they really meant was that they had a credible instrument: a variable that breaks the link between treatment assignment and the potential outcomes for some or all of the units. However, the lead PA article places difference-in-differences, regression discontinuity, and matching methods under the tent of natural experiments. While I like (and use) these techniques and find them compelling, only some of them explicitly rely on an IV-type argument. Maybe I have more to learn.

The problem with any randomization that isn't controlled by the researcher is that extreme skeptics like me can then try to spin complicated stories about how confounding could occur. This is what I found myself doing while reading Guns, Germs, and Steel. An extremely simplified version of Diamond's argument is that geography, not genetics, determines which human societies become dominant and which are conquered or destroyed. He devotes the entirety of chapter 2 to discussing the settlement of Polynesia by people who come from essentially the same genetic stock but experienced different geographies once they settled particular islands. The random variation in geography is interpreted as the cause for significant variation in the trajectories of the peoples of each island or group of islands.

This might be a natural experiment if Diamond could show that people were somehow randomly assigned to different islands. The problem is that different types of people might chose to live on different islands. Although it may be random which islands an exploratory party reaches, the explorers can choose to stay or move on for reasons that might be related to genetic variation. Similarly, explorers and colonists are probably not a random sample of the population, so the types of people that reach a far off island might have different genetic traits than those that remain in already established population centers. You get the idea.

I should reiterate that these reservations are just my gut reactions rather than a well thought-out assault on the use of natural experiments. I'm interested to read more: Jared Diamond and our very own James Robinson have a new book out on the subject that I'm excited to read. Thad Dunning has written on the topic, as have others.

Bottom line: I'm thrilled (and jealous) whenever social scientists find some plausibly exogenous variation to exploit for causal inference. I think it should happen more. I just worry that by attaching the "experimental" label to these studies, we endow them with undue credibility.

Posted by Richard Nielsen at 11:30 AM | Comments (4)

January 19, 2010

Bike helmet laws

I'm into biking (mostly road-biking these days) so I was interested to read a post on the New York Times' "Freakonomics" blog about a study that uses variation in bike helmet laws across US states to show that helmet laws decrease bike riding among kids and teens. Since I think that most people should ride bikes most of the time AND I have been known to bug people to wear helmets, perhaps I've been working against myself.

A few things came to mind while reading the study. First, the study shows that helmet laws have an effect on bike safety for kids in the same age ranges. Unless I missed something, it seems like part of this effect could be due to fewer kids riding bikes (in addition to the obvious safety improvement that comes from actually wearing a helmet). I'd be curious how much the decrease in bike use is influencing the increase in safety, especially if kids are simply deciding to do other things like skateboarding that are perhaps equally dangerous but don't require helmets (a possibility mentioned by the authors). This may mean that the total effect of helmet laws on child safety is less than the effect estimated in the paper because some of the decreases in bike injuries are counter-balanced by increases in other types of injury that aren't part of the study.

Second, the authors use some fixed effects and diff-in-diff models, but I think this paper is calling out for the synthetic control method developed by Abadie, Diamond, and Hainmueller. The policy intervention is clean and there are a reasonable number of states that don't have laws, so building synthetic matches might be feasible. There might be some interference problems with states that pass helmet laws later, but those are details...

I'll end this post with a shameless plug: bike more! (and wear a helmet)

Posted by Richard Nielsen at 9:01 PM | Comments (3)

December 22, 2009

Happy Holidays?

This morning the New York Times alerted me to a Science piece written by two economists working on measuring happiness. Their basic finding is that objective measures of quality of life (nice climate, etc) are pretty highly correlated with subjective, self-reported measures of how satisfied people are with their lives. They provide a ranking of US states by happiness level, accessible here, which shows Louisiana first and New York last, with Massachusetts falling to 43rd. Go figure -- I like living in MA.

I really want to see some cross-national comparisons but I doubt anyone will be moving on to that unless the World Bank picks up Bhutan's Gross National Happiness measure as one of their development indicators.

Happy holidays to all!

Posted by Richard Nielsen at 9:25 AM | Comments (3)

November 28, 2009

Everything about causal inference in 40 pages

Judea Pearl describes his new article Causal inference in statistics: An Overview as "a recent submission to Statistics Survey which condenses everything I know about causality in only 40 pages." That seemed like a bold claim, but after reading it I'm sold. I don't come from Pearl's "camp" per se, but I found this a really impressive overview of his approach to causation. His overtures to folks like me who use the potential outcomes framework were much appreciated, although it is clear throughout that there is still intense debate on some of the issues. The bottom line: if you've ever wondered what the structural equation modeling approach to causal inference is all about, this is your one-stop, must-read introduction (and an insightful, engaging, and thorough one at that).

Posted by Richard Nielsen at 10:54 AM

November 11, 2009

Answering "why" questions

Brandon Stewart pointed me to an interesting blog post by Andrew Gelman that touches on the issue of explaining the "causes of effects." The basic point is that "why" questions are difficult to answer in a potential outcomes framework but often we really care about them. Some folks in political science have gone so far as to argue that researchers using "qualitative" methods are more inclined (and better able) to tackle these "why" questions than their "quantitative" colleagues who mostly focus on "effects of causes."

This has been on my mind lately -- as part of a class in the statistics department, I've had several conversations with Don Rubin about how retrospective "case-control" studies might fit into the potential outcomes framework. The goal of the medical researchers that execute these studies is usually a "why" question: why did an outbreak of rare disease X occur, which genes might cause breast cancer, etc. Case-control studies and their variants are great for searching over a number of possible causes and pulling out the ones that have strong associations with the outcome, but they aren't so great for estimating treatment effects. Rubin suggests that the proper way to proceed is probably to first use a case-control study to search over a number of possible causes and then estimate treatment effects for the most likely causes using a different sampling method (matched sampling for situations where the research has to be observational, experimentation when it's possible). It seems like this already happens to some extent in biostatistics and epidemiology and it also happens informally in political science.

I think this formulation suggests that answering a "why" question requires both "causes of effects" and "effects of causes" approaches; we need to search over a number of possible causes to identify likely causes, but we also need to test the effectiveness of each likely cause before we can say much about the causal effect. We probably still can't answer questions like "what caused World War I" but maybe this gets us somewhere with more tractable types of "why" questions.

Posted by Richard Nielsen at 4:30 PM

October 30, 2009

Happy Halloween

It made my day when this showed up in my inbox this morning. I'm glad to see someone knows what to do if/when the zombie outbreak occurs.

Posted by Richard Nielsen at 10:05 AM

October 21, 2009

Multiple Instruments

I recently found a paper by Angus Deaton that attempts to (1) discount the usefulness of instrumental variables for making causal inferences in development economics and (2) discount the usefulness of field experiments. He has definitely stirred the pot a little and is now part of an interesting debate, although the discussion seems to be more focused on Deaton's controversial claims about experiments.

I think Deaton overlooks some of the benefits of experimental research but his criticism of instrumental variables seems dead on, especially on the use of multiple instruments (see pages 12-13). Intuitively, we might think that having many instruments makes for better causal inference -- if one doesn't work out, then the others will pick up the slack. Following this logic, studies that use multiple instruments and "test" for exogeneity with overidentification tests have become popular in the instrumental variables literature. Essentially, these tests boil down to re-estimating the model with subsets of the instruments and showing that the estimated coefficients don't change dramatically. This can mean one of two things: (a) not just one, but all of the instruments are exogenous, or (b) not just one, but all of the instruments are endogenous. Personally, I think the probability of finding even a single good instrument for a given problem is small, so when shown a research design with multiple instruments, I need some serious convincing that miraculously all of the instruments are valid.

I am probably overly skeptical and I am very sympathetic to heroic attempts to solve difficult problems of causal inference to answer important questions. Still, it seems that having multiple instruments can become an embarrassment of riches. A good instrument is so hard to come by that having too many starts to lend evidence against an empirical argument rather than for it.

Posted by Richard Nielsen at 12:41 PM