Social Science Statistics Blog

I think this entry is a good statement of where the literature on matching is, but I think almost all of the literature has this point wrong. Hypothesis tests for checking balance in matching are in fact (1) unhelpful at best and (2) usually harmful.

Suppose you had a control group and a treatment group that are identical (exactly matched) except for one person, or except for a bunch of people in one very minor way. Suppose hypothesis tests indicate no difference between the groups, and so you'd be in the situation of reporting balance was great and no further adjustment was needed. (We might think of this as a real experiment where the outcome variable hasn't been collected but is expensive to do so.) If you were given the chance of dropping the one or few people that caused the two groups to differ and replacing them with others that exactly matched, would you do so? Since the dimension on which the inexact match or matches occurred might be the one that has a huge effect on your outcome variable, the bias due to not switching could be huge. So you'd undoubtedly make the switch, despite the fact that the hypothesis test indicated that there was no problem. Hence (1) the tests are unhelpful: passing the test does not necessarily protect one from bias more than failing the test.

Now suppose you have data that don't match very well by all hypothesis tests and you randomly (rather than systematically to improve matching) drop observations, in a bad application of matching. what will happen? Your t-tests or ks-tests or any other hypothesis tests will lose power and so will indicate that balance is getting better and better. yet, bias is not changing at all, and efficency is dropping fast. The tests are telling you to discard data! Hence (2) hypothesis tests to evaluate balance are harmful, quite seriously so.

The fact is that there is no superpopulation to which we need to infer features of the explanatory variables; all analysis models we regularly use after matching are conditional on X. Balance should be assessed on the observed data, and not be the subject of inference or hypothesis tests.

This message rehearses an argument in a to-be-revised version of our matching paper (by Ho, Imai, King, and Stuart at http://gking.harvard.edu/preprints.shtml#matchp) that we hope to be finished with and post in a couple of weeks.

Gary King

Hi Jens,

The question of "When have we achieved balance" goes right to the heart of the matter. As you note, Diamond and Sekhon (2005) makes the case that prior balance tests utilized in previous work have been too lenient. (Not surprisingly: After all, why should it be easy to make reliable causal inferences in non-experimental settings?) In particular, we found that all prior work over the past decade with the canonical LaLonde/Dehejia Wahba dataset failed to achieve a degree of balance adequate to ensure reliable results. My own view is that this problem is not limited to the LaLonde/Dehejia Wahba datasets, but is in fact endemic to > 95% of all matching-based analyses across the disciplines. In the LaLonde/Dehejia-Wahba setting, you don't begin to see good results until the lowest paired t- and KS-tests p-value across ALL covariates and 2-way interactions is greater than 0.15 to 0.2. (Incidentally, this very high p-value threshold was Jas Sekhon's intuition very early on, prior to us doing the work and validating it.) How many prior matching analyses can claim to have achieved this level of balance? None that I know of. That's the bad news (for prior work). The good news (for future work, and aspiring muckraking graduate students) is that this level of balance is achievable in many datasets, if you and your computer work hard enough and use the right methodology (ie., search over the space of covariate weights a la Diamond and Sekhon 2005). Cheers.

What's the cite for Diamond and Sekhon 2005?

Hi Alexis,

thanks again for your muckraking comment. I was actually pretty stunned when I saw your evidence on what balance standards are actually needed to make the inferences in the LaLonde dataset somewhat reliable. That’s a balance hurdle that many articles do not jump over.

It would be interesting to know whether the same pattern holds in other experimental datasets where we can do LaLonde type evaluations by integrating observational data. If we find that the required balance levels at which we get somewhat reliable answers is similarly high this would start to make me anxious about a lot of the published matching results out there.

Did you try that yet?

Best,
Jens

The citation is:

Alexis Diamond and Jasjeet Sekhon. 2005. Genetic Matching for Estimating Causal Effects: A General Multivariate Matching Method for Achieving Balance in Observational Studies.

You can download the paper at:
http://sekhon.polisci.berkeley.edu/papers/GenMatch.pdf

Jens, your comment about investigating other LaLonde-like projects is right on the mark, and we have a graduate student working on it. You and the rest of the blog community will be among the first to know what we find out.
Of course, there have been a number of studies that have attempted to answer LaLonde-like questions. See the text "Learning More From Social Experiments" by Harold Bloom (an early innovator of IV-estimation) for more information.