Social Science Statistics Blog: Did You Achieve Balance?! Part I

May 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

This blog makes public the hallway conversations about social science statistical methods and analysis from the Institute for Quantitative Social Science and related research groups. Expect to see posts on trends in methodological thought, questions and comments, paper and conference announcements, applied problems needing methodological solutions, and methodological techniques seeking applied problems. Also included are summaries of papers and comments from a popular weekly research workshop held here and billed as a tour of Harvard's statistical innovations and applications with weekly stops in different disciplines. [read more...]

« Applied Statistics - Gopi Goswami | Main | New IR Data Set with 10 Million Dyadic Events »

25 October 2005

Did You Achieve Balance?! Part I

Jens Hainmueller

There exists a growing consensus in the causal inference literature that when it comes to bias adjustment under selection on observables, matching methods dominate ordinary regression (esp. when discrepancies between groups are large). But how do we judge the quality of a matching? My professors tell me: "We want good balance." Sounds great, so I thought at first. Reading more matching articles, however, I soon became somewhat startled by the scholarly disagreement about what actually constitutes "good" balance in observational studies. Despite the fact that matching methods are now widely used all across the social sciences, we still lack shared standards for covariate balance: Which tests should be used in what type of data? What are their statistical properties and how do they compare to each other? And how much balance is good enough?

From reading this literature (sincere apologies if I have missed something relevant), it seems to me that most people agree that paired t-tests for differences in means are obligatory. T-tests are useful because matching by construction produces matched pairs. But should we test by comparing whole groups (treated vs. matched-untreated) or within propensity score ("PS") subclasses? A problem with the latter may be that the choice of intervals can be arbitrary, which is critical as interval width affects the power of the test (Smith and Todd 2005).

Moreover, which covariates should we t-test balance on? At least all that are included in the matching (right?), but how about other moments, the full set of interactions and higher-order terms, etc? The latter seems helpful to minimize bias but is done once in a blue moon (at least in the papers that I encountered). Most authors avoid these additional tests since they exacerbate common support problems and substantially raise the hurdle for obtaining balance.

Finally, should we t-test balance on the PS score and or the covariates othorgonalized to the PS score? How do we deal with the estimation uncertainty in these variables? And what does it mean -- as happens sometimes in practice -- to have remaining imbalance on the PS while all covariates are balanced?

Stand by for part II of this post tomorrow.

Posted by James Greiner at October 25, 2005 5:00 AM

Comments

Hi Jens,

Great posting. One question-- you write that "from reading this literature... most people agree that paired t-tests for differences in means are obligatory." I was under the impression (perhaps mistaken) that Diamond and Sekhon (2005) was one of only a few papers arguing for the paired version of this test, and that most analysts run the unpaired version of the t-test. From your reading of the literature, what evidence did you find showing consensus support for the paired version of the test?

Posted by: Alexis Diamond at October 26, 2005 4:21 PM

Hi Alexis,

thanks for your comment. You're right: appetitus rationi pareat!! Please take my apology that using the word *paired* here was driven more by desire than reason.

What I meant to say, of course, was that most people agree that *unpaired* t-tests for differences in means are obligatory. From my reading this is the most commonly used balance test.

I salute to your plea for using paired t-tests, and I agree that there isn't a consensus on this. The converse is probably true, they are almost never used; despite the fact that they make sense, since matching produces matched pairs.

Cheers,

Jens

Posted by: Jens at October 26, 2005 6:46 PM

Jens,

You're taking the Rubin/Imbens statistics course this semester, aren't you? What is the word from on high? These guys are two of the great gurus of this field. What are they teaching the next generation of matching methodologists? How do they answer your question? I hope we find out in your next blog posting, "Did You Achieve Balance, part III". Looking forward to it :)

Posted by: Alexis Diamond at October 26, 2005 7:43 PM

Social Science Statistics Blog

Authors' Committee

Chair:

Members:

Weekly Research Workshop Sponsors

Recent Comments

Recent Entries

Categories

Blogroll

Archives

Notification

Did You Achieve Balance?! Part I

Comments

Notification

Post a comment