Social Science Statistics Blog

« October 16, 2006 | Main | October 19, 2006 »

18 October 2006

The probability of vanishing posts

There was for a while a post on this blog with comments about an article on deaths in the Iraq war. Despite many good points all parties had in this discussion, it was distracting some folks here from our mission to make the world safe for quantitative methodology. If you're interested in reading more about this subject, we recommend Andrew Gelman's blog post on this subject, which includes many of those who posted and commented here. Thanks for the original post and everyone who commented, and sorry for the confusion.

Posted by Gary King at 4:20 PM

Meta-analysis: To Trust or Not to Trust

Cassandra Wolos

Social scientists, who often have a limited ability to create true experiments and replicate studies, value ways to learn from the synthesized results of previous work. A popular quantitative tool designed for this purpose is meta-analysis, which calculates a standardized effect size for each of a set of studies in a literature review and then performs inference on the resulting set of effect sizes. Meta-analysis is particularly common in education research.

Can we trust the results of these analyses?

On the one hand, when performed correctly, meta-analysis should successfully summarize the information available in multiple studies. Combining the results in this way can increase the power of overall conclusions when the sample size in each study is relatively small.

On the other hand, a good meta-analysis relies on the assumption that the original studies were unbiased and generally well-performed. In addition, we hope that the researchers in each study had the same target population in mind and worked independently of each other. Further complicating matters is the potential for publication bias – a meta-analysis will rarely include unpublished studies with less impressive effect sizes.

The second hand represents the view of Derek Briggs at the University of Colorado, Boulder, who in a 2005 Evaluation Review paperobjected to what he saw as the overuse of meta-analysis in social science research. He also suggested that assumptions necessary for a reliable meta-analysis are not always met.

More to come on this topic next time.

Posted by Cassandra Wolos at 10:00 AM