Social Science Statistics Blog

I am in fact deeply troubled (though not much surprised) by the Gerber and Malhotra report. I grant that your benign explanations all could be true. But I just don't believe that they account for much. Short of flat out ying about their z-scores ("hey, what's 1.94 against 1.98?") I rather doubt that many political scientists - or any social scientists - are going out to collect more data, it's just not the thing we can do most of the time. Much more likely, they are practicing self-deception via ill-applied regression diagnostics: hunting down outliers that influence the results whether or not these cases actually belong in the analysis; eliminating small subsets of cases that make a difference because no reviewer will know; omitting one or two control variables, trimming a year from a long panel; messing with missingness; using some seemingly nifty but deep down inapporpriate way of "adjusting" standard errors for imagined offenses etc.

In general, I think that the publication bias problem is far more significant (if you'll pardon the expression) in largely observational sciences than it is in the experimental context that you are describing. Leaving aside the question of the appropriateness of hypothesis testing in general (something that I suspect we will hear more about on this blog in the future), in the experimental sciences there is the possiblity of true replication (as in generating new data, not just successfully reanalyzing existing data) to weed out the cases where the experimenter got lucky. We usually don't have the same good fortune in observational sciences - we only get to run the world once, so it becomes difficult to sort out the "lucky" results from the rest.

Hmm, I can definitely see what you mean: if one doesn't (by and large) run experiments, then my possible explanation for why there are so many studies just over significance doesn't apply. The null result explanation, though, still seems valid to me (though it might occur in the context of other, more undesirable, reasons for having fewer null results).

I guess the authors argue that even in the case where you've run an experiment and then added a few more N in order to "push it over" to p>0.05, this is unfortunate because -- in combination with the "file drawer" problem -- it makes doing meta-analyses problematic (you aren't truly "randomly sampling" from results space). That's true, but I think meta-analyses are often problematic for many other reasons also. :)

On reported statistical tests being too
close to significance due to rounding,
see the paper by Diaconis and Freedman,
On Rounding Percentages,JASA, vol. 74, 359-364