Social Science Statistics Blog

James Greiner writes: Perhaps more broadly, what is the theory that tells me that the genetic algorithm search is attempting to do the right thing (and what is that "right thing")?

This is a good question. Just to be clear, let's get the GA part out of it. There is very good theory that a GA will search a space at least well as randomization and if there is weak connective information between points (this assumption is weaker than continuity) it will out perform randomization, usually by a polynomial factor (i.e., swamp randomization).

Now to the main point. There are some distributions for which testing the margins and x-way interactions is good enough, but for most distributions this is not enough! This is why there will always be an art to this. Also, one can use multivariate KS tests which test the complete joint distribution, but these methods will require a lot of data. But there is a theoretical path here.

James Greiner writes: (I have heard that K-S in particular has low power)

For some null hypothesis, but for others it is okay. That's why we advocate that a researcher use a LARGE variety of functions of the data because, as we note, different functions will have different power against different departures from balance. For example, if the distributions just differ in the means, than a paired t-test is much greater power. For differences related to skew and kurtosis, the KS test is better than the t-test. For differences in the tails, the Wilcoxon Mann-Whitney test is generally more powerful than the KS test.

My code will use the Mann-Whitney test if that is passed to it instead of the KS. But it doesn't make much difference because the p-value that comes out of genmatch is IRRELEVANT. As the LaLonde example shows (see figures in Genetic Matching), the conventional test levels are no where near enough to achieve a reliable inference. So all one just needs a metric to move along on. And for that the KS and Mann-Whitney amount to basically the same thing.

James Greiner writes: I have only one suggestion on this
subject: whatever method one uses to create matched sets of treated
and control groups, after all ordinary checking of marginal
distributions is complete, throw something completely wild at the
results. For both groups, calculate a fifth moment of covariate one,
interact it with a third moment of covariate two and a second moment
of covariate three. Do a test and see what happens. If the two
groups have the same joint distribution of their covariates...

I think this is a very good idea and is what I currently recommend. For example, in my Matching Software, after genetic optimization, it is suggested that one use the MatchBalance function and through in some new function of X to test, just as you suggest.

Finally, I will note that GenMatch is agnostic as to what precise fit criterion is used. One could, for example, work with features of the QQ plots directly or a multivariate histogram.

Jim Greiner writes: "Is the objective in matching treated and control units to find sets of observations with the same JOINT distribution of the covariates, which is what one would have in a randomized experiment?" This is *not* what one has in a regular randomized experment in practice. In fact, after GenMatch-ing (in the DW case, for example), treated and GenMatch-ed control units have more similar distributions of covariates than did the the original randomized experimental sample. In practice, Genetic Matching can and often will produce better balance **on observables** than a randomized experiment.

A larger point: This kind of statistical work is, in my view, largely about making your argument coherently, stating your assumptions clearly, and letting the scientific community judge. Assuming we all agree on how to intepret the identifying assumptions a method requires, then the role of the analyst is to muster evidence that these assumptions have been sufficiently satisfied. In OLS, one has to argue that there are no omitted variables. In IV, you've got to make the case for exclusion. And in matching, you have to presume conditional independence. And in practice, in all these cases and in all others I know of (except maybe bounds analysis, sometimes), the analyst that claims all assumptions perfectly obtain is almost always stretching the truth in front of a knowing audience that is supposed to know how to assess the consequences. OLS is unbiased under certain conditions, but nearly all OLS-based estimates are certainly biased. Ditto for IV, and everything else including matching. The question, is the bias going to matter?
So, Jim asks about joint distributions. If that's what it takes to be convincing, check all three-way interactions and beyond.
And by the way, when we are considering how these sorts of arguments are made in our community, we should ask ourselves--does the analyst's method encourage honest or dishonest analysis? Methods (like matching) that do not require the use of outcomes encourage honesty. Methods like regression that involve running analyses, getting an answer, and repeating until JACKPOT... that's a different story altogether.