Social Science Statistics Blog

November 11, 2009

Answering "why" questions

Brandon Stewart pointed me to an interesting blog post by Andrew Gelman that touches on the issue of explaining the "causes of effects." The basic point is that "why" questions are difficult to answer in a potential outcomes framework but often we really care about them. Some folks in political science have gone so far as to argue that researchers using "qualitative" methods are more inclined (and better able) to tackle these "why" questions than their "quantitative" colleagues who mostly focus on "effects of causes."

This has been on my mind lately -- as part of a class in the statistics department, I've had several conversations with Don Rubin about how retrospective "case-control" studies might fit into the potential outcomes framework. The goal of the medical researchers that execute these studies is usually a "why" question: why did an outbreak of rare disease X occur, which genes might cause breast cancer, etc. Case-control studies and their variants are great for searching over a number of possible causes and pulling out the ones that have strong associations with the outcome, but they aren't so great for estimating treatment effects. Rubin suggests that the proper way to proceed is probably to first use a case-control study to search over a number of possible causes and then estimate treatment effects for the most likely causes using a different sampling method (matched sampling for situations where the research has to be observational, experimentation when it's possible). It seems like this already happens to some extent in biostatistics and epidemiology and it also happens informally in political science.

I think this formulation suggests that answering a "why" question requires both "causes of effects" and "effects of causes" approaches; we need to search over a number of possible causes to identify likely causes, but we also need to test the effectiveness of each likely cause before we can say much about the causal effect. We probably still can't answer questions like "what caused World War I" but maybe this gets us somewhere with more tractable types of "why" questions.

Posted by Richard Nielsen at 4:30 PM | Comments (1)

October 30, 2009

Happy Halloween

It made my day when this showed up in my inbox this morning. I'm glad to see someone knows what to do if/when the zombie outbreak occurs.

Posted by Richard Nielsen at 10:05 AM | Comments (6)

October 21, 2009

Multiple Instruments

I recently found a paper by Angus Deaton that attempts to (1) discount the usefulness of instrumental variables for making causal inferences in development economics and (2) discount the usefulness of field experiments. He has definitely stirred the pot a little and is now part of an interesting debate, although the discussion seems to be more focused on Deaton's controversial claims about experiments.

I think Deaton overlooks some of the benefits of experimental research but his criticism of instrumental variables seems dead on, especially on the use of multiple instruments (see pages 12-13). Intuitively, we might think that having many instruments makes for better causal inference -- if one doesn't work out, then the others will pick up the slack. Following this logic, studies that use multiple instruments and "test" for exogeneity with overidentification tests have become popular in the instrumental variables literature. Essentially, these tests boil down to re-estimating the model with subsets of the instruments and showing that the estimated coefficients don't change dramatically. This can mean one of two things: (a) not just one, but all of the instruments are exogenous, or (b) not just one, but all of the instruments are endogenous. Personally, I think the probability of finding even a single good instrument for a given problem is small, so when shown a research design with multiple instruments, I need some serious convincing that miraculously all of the instruments are valid.

I am probably overly skeptical and I am very sympathetic to heroic attempts to solve difficult problems of causal inference to answer important questions. Still, it seems that having multiple instruments can become an embarrassment of riches. A good instrument is so hard to come by that having too many starts to lend evidence against an empirical argument rather than for it.

Posted by Richard Nielsen at 12:41 PM | Comments (2)