Joris Frese (Applied Stats Workshop)
Date and Time
Location
Speaker & Title
Joris Frese (European University Institute), "Type I Error Inflation in Unexpected Event During Survey Designs"
Abstract
The ‘Unexpected Event During Survey' design (UESD) leverages quasi-random timing of interviews relative to external events for causal identification. In this research note, we empirically evaluate the UESD's Type I (false positive) error rate using large-scale survey data covering 40 countries between 2002 and 2024. Based on over 42 million placebo tests, our analysis reveals that standard inferential approaches yield false positives at rates more than double the conventional 5% threshold, indicating miscalibrated p-values and potentially erroneous conclusions. We further document significant heterogeneity in this error inflation across countries, surveys, outcome variables, and model specifications. To address this, we propose a randomization-based adjustment procedure tailored to context-specific false positive rates and provide open-source software for its implementation. We illustrate our approach by replicating the two most highly cited UESD articles published in the APSR.
The Applied Statistics Workshop (Gov 3009) meets all academic year on Wednesdays. This workshop is a forum for advanced graduate students, faculty, and visiting scholars to present and discuss methodological or empirical work in progress in an interdisciplinary setting. The workshop features a tour of Harvard's statistical innovations and applications with weekly stops in different fields and disciplines and includes occasional presentations by invited speakers.
All interested Harvard affiliates are invited to attend. Lunch will be provided.