Social Science Statistics Blog

« Imai on "Identification and Inference in Causal Mediation Analysis" | Main | Interest in computer science is volatile »

10 November 2008

Conditions under Which Observational Studies Produce Comparable Causal Estimates

In the latest issue of Journal of Policy Analysis and Management, Thomas Cook, William Shadish and Vivian Wong wrote a paper proposing three conditions under which experiments and observational estimates are comparable based on their review of 12 recent within-study comparison studies. It is a little bit confusing, at least to me at first glance, to use "conditions" rather than "designs" here, as what the authors are really arguing is under three different types of research designs estimates from observational studies are comparable to causal estimates. More specifically, they suggest that:

1) regression discontinuity (RD) estimator produces similar effect estimates to experimental ones;

2) when intact group matching is used to minimize pre-test differences in at least outcome measures between the experiment and comparison populations, estimates from observational studies are trustworthy; and

3) when selection process into treatment is completely or plausibly known and could be properly measured, statistical procedures like propensity score matching can provide unbiased estimates.

As you can see, these three claims are based on selected published or to-be-published studies. But publication bias may lead them to overstate these claims, which in this case means observational studies with estimates comparable to experimental ones are disproportionally likely to be published than those without comparable estimates, and so how accurately or confidently we can rely on these claims to evaluate the comparability of estimates from observations studied remains ambiguous. In addition, this issue also relates to what standards we are using to judge comparability. If the standards are fuzzy, our judgment will be fuzzy to some extent as well. But overall, I appreciate the authors' enormous efforts on tracing recent literature on this topic and the resulted paper is full of wisdom.

When I am finishing this post, I realize that this paper was actually presented at our applied statistics workshop last October. But here comes the official version of the paper. And I think this is a very important topic that is worth a revisit.

Source:
Thomas Cook, William Shadish and Vivian Wong. 2008. "Three Conditions under Which Experiments and Observational Studies Produce Comparable Causal Estimates: Findings from Within-Study Comparisons", Journal of Policy Analysis and Management, Vol. 27, No 4, 724-750.

A previous paper distributed at the applied statistics workshop:
http://www.iq.harvard.edu/blog/sss/archives/2007/10/tom_cook_on_whe.shtml

Posted by Weihua An at November 10, 2008 11:23 AM