David Ham (Workshop in Applied Statistics)

Date: 

Wednesday, February 15, 2023, 12:00pm to 1:30pm

Location: 

CGIS Knafel Building, room K354 or Online via Zoom

Today's Speaker

David Ham (Harvard), "Design-Based Confidence Sequences for Anytime-valid Causal Inference"

Abstract

Many organizations run thousands of randomized experiments, or A/B tests, to statistically quantify and detect the impact of product changes. Analysts take these results to augment decision-making around deployment and investment opportunities, making the time it takes to detect an effect a key priority. Often, these experiments are conducted on customers arriving sequentially; however, the analysis is only performed at the end of the study. This is undesirable because strong effects can be detected before the end of the study, which is especially relevant for risk mitigation when the treatment effect is negative. Alternatively, analysts could perform hypotheses tests more frequently and stop the experiment when the estimated causal effect is statistically significant; this practice is often called "peeking." Unfortunately, peeking invalidates the statistical guarantees and quickly leads to a substantial uncontrolled type-1 error. Our paper provides valid confidence sequences from the design-based perspective, where we condition on the full set of potential outcomes and perform inference on the obtained sample. Our design-based confidence sequence accommodates a wide variety of sequential experiments in an assumption-light manner. In particular, we build confidence sequences for 1) the average treatment effect for different individuals arriving sequentially, 2) the reward mean difference in multi-arm bandit settings with adaptive treatment assignments, 3) the contemporaneous treatment effect for single time series experiment with potential carryover effects in the potential outcome, and 4) the average contemporaneous treatment effect in panel experiments. We further provide a variance reduction technique that incorporates modeling assumptions and covariates to reduce the confidence sequence width proportional to how well the analyst can predict the next outcome.

The Applied Statistics Workshop (Gov 3009) meets all academic year, Wednesdays, 12pm-1:30pm, in CGIS K354. This workshop is a forum for advanced graduate students, faculty, and visiting scholars to present and discuss methodological or empirical work in progress in an interdisciplinary setting. The workshop features a tour of Harvard's statistical innovations and applications with weekly stops in different fields and disciplines and includes occasional presentations by invited speakers.

More information is available at the Gov 3009 website: https://projects.iq.harvard.edu/applied.stats.workshop-gov3009

All interested Harvard affiliates are invited to attend.