Social Science Statistics Blog

having worked with nsw data, i agree we need alternatives that permit direct comparison of experimental/scientific versus statistical/observational solutions to the problem of inference. that said, tests based on "messy" data sets such as supported work seem especially appropriate for social scientists modeling messy processes. we just need more tests under different conditions. fortunately, i found 121 disparate data holdings at icpsr under keyword "experimental." we don't have to keep hammering on supported work.

Hi Chris Uggen,

thanks for your comment.

I agree that tests based on "messy" data sets such as supported work seem especially appropriate for social scientists modeling messy processes. Yet, I think the question remains: if we do not get it right in messy data, what do we conclude? Is it that our estimators are really ill-equipped to correct the bias we could potentially adjust for given the covariates we have in the data, or is it that our estimators actually do quite well, but hidden bias is too large in which case we would still get it wrong even when we perfectly adjusted for observables because there remains imbalance on unobserved stuff that is strongly correlated to both treatment assignment and the outcomes of interest. In principle, bias adjustment properties and the issue of hidden bias are too distinct problems, but in the Lalonde example they are both enmeshed. Of course, one could argue that in a particular application, if estimators A get it right despite hidden bias while estimator B doesn’t, then A is clearly preferred over B…

Best,
Jens

I think the LaLonde data are a bad data set to be using as a test data set for all sorts of procedures for three reasons.

First, the relevant sample size is really small. There are not very many participants at all and, as Dehejia and Wahba show clearly that the number of non-participants who look anything like the participants is quite small (which is why matching without replacement is a disaster in these data).

Second, the evidence in Smith and Todd (2005), plus even an epsilon of thinking about the nature of the selection in this context, make it clear that the unconfoundness assumption (called the conditional independence assumption in economics) does not hold in these data. Remember that the participants here are ex-cons, ex-addicts and dropouts. No variables are available in the comparison group data sets to identify non-participants who are ex-cons or ex-addicts. We have strong priors that ex-cons and ex-addicts are different than other folks conditional on the age, race, schooling and mismeasured annual earnings, which are the only conditioning variables on offer. Much as I respect Bob LaLonde, it was silly, in retrospect, to think that this would ever work.

Third, the data do not contain any good instruments to use to examine the efficacy of either the bivariate normal selection model or standard IV methods. The exclusion restrictions employed in the LaLonde paper would not pass the laugh test in a seminar these days. In addition, as we note in Smith and Todd (2005), his bivariate normal results are wrong because he does not take account of choice-based sampling (to which the estimator is not robust) and because one of his exclusion restrictions is actually a perfect one-way predictor of treatment status.

Finally, it is important to frame the question correctly. The question is not "does matching work" or "does IV work". Both methods work when their identifying assumptions hold in the data and do not work otherwise. The question we should be asking is, in what contexts, defined by data and institutions, does each method work well. I would argue that the NSW data commonly used for these studies does not satisfy the identifying assumptions for any of the commonly used estimators, which is why I think it is time to stop using it as a canonical data set.

Jeff Smith

to second Jeff's comments, i too think the NSW is a very limited dataset. i also think we should try to apply LaLonde's (86) idea to other, better datasets. in the Diamond/Sekhon (2005) paper, we show Monte Carlo results that represent a significant step away from NSW and into something new.

the NSW doesn't provide a very rich set of covariates at all. i think that's why this dataset requires an extremely high degree of balance (as measured by the leximin p-value) before the simple ATT difference-of-matched-means estimate begins to converge on the experimental result. Once you balance super-duper well on the NSW's Xs and all their quads and 2-way interactions, you're probably balancing on a bunch of other stuff you haven't measured. at least that's my conjecture for what's going on in this case.

but yes, let's move on to other datasets and new Monte Carlos, please! :) and let's use these new analyses to validate current notions of balance tests and balance thresholds and expectations for reliability in observational studies.

alexis