Social Science Statistics Blog

« October 2009 | Main

21 November 2009

Violations of SUTVA

Network methods and methods for causal inference are popular areas of research in social sciences. Often they are considered separately due to a fundamental difference in their basic assumptions. Network methods assume that individual units are interdependent, that one network member's actions have consequences for other members of the network. Methods for causal inference, in contrast, often rest on the Stable Unit Treatment Value Assumption (SUTVA). SUTVA requires that the response of a particular unit depends only on the treatment to which he himself was assigned, not the treatments of others around him. It is a useful assumption, but as with all assumptions, there are circumstances in which it is not credible. What can be done in these circumstances?

When researchers suspect that there may be spillover between units in different treatment groups, they can change their unit of analysis. Students assigned to attend a tutoring program to improve their grades might interact with other students in their school who were not assigned to the tutoring program and influence the grades of these control students. To enable causal inference, the analysis might be completed at the school level rather than the individual level. SUTVA would then require no interference across schools, a more plausible assumption than no interference across students. However, this approach is somewhat unsatisfactory. It generally entails a sharp reduction in sample size. More importantly, it changes the question that we can answer: no longer can we learn about the performance of individual students, we can only learn about the performance of schools.

I have not come across a more satisfactory statistical solution for circumstances in which SUTVA is violated. In an interesting new paper, Manski provides some bounds on treatment effects in the presence of social interactions. Unfortunately, these bounds are often uninformative, since when SUTVA is violated random assignment to treatment arms does not identify treatment effects. Sinclair suggests using multi-level experiments to empirically identify spillover effects. This approach (which relies on multiple rounds of randomization to test if treatment effects are overidentified, as we would expect if there were no spillovers) is appealing, as the process of diffusion within networks is of great scientific interest. However, it does not help identify treatment effects when spillovers are present. Neither can we simply assume that effects estimated under SUTVA represent upper bounds on the true effects, because it is possible that interference across units intensifies the treatment effects rather than diluting them. Manski's paper seems like a useful foray into an open area of research. Let me know of other work on methods for causal inference in network-like situations where interference across units is likely.

Posted by Deirdre Bloome at 4:50 PM | Comments (1)

17 November 2009

Dynamic Panel Models

I have been toying around with dynamic panel models from the econometrics literature and I have hit my head up against a key set of assertions. First, a quick setup. The idea with these models is that we have a set units which we measure at different points in time. For instance, perhaps we survey a group of people multiple times in the course of an election and ask them how they are going to vote, do they plan to vote, how do they rate the candidates, etc. We might then want to know how these answers vary over time or with certain covariates.

Here is a typical model:

There are two typical features of these models that seem relevant. First, most include a lagged dependent variable (LDV) to account for persistence in the responses. If I was going to vote for McCain the last time you called, I'll probably still want to do that this time. Makes sense. Second, we include a unit-specific effect, alpha, to account for all other relevant factors. Dynamic panel models tend to identify their effects with a simple differencing by running the following model:

Which eliminates the unit-specific effect by the differencing, but our parameters remain, ready to be estimated. I should note that there are some identification issues left to solve and the differences between estimators in this field mostly have to do with how to instrument for the differenced LDV.

Reading these models, I have two questions. One, is there a reason to expect that we need both a LDV and a unit-specific effect? This means that we expect that there is a shock to a unit's dependent variable that is constant across periods. I find this a strange assumption. I understand a unit-specific shock to the initial level and then using LDV thereafter, but in every period?

Two, the entire identification strategy here is based on the additivity of the model, correct? If we were to draw a directed acyclic graph of these models, it would be trivially obvious that we could never identify this model nonparametrically. I understand that we sometimes need to use models to identify effects, but should these identifications depend so heavily on the functional form? It seems that this problem is tied up in the first. We are allowing for the unit-specific effect as a way to free the model of unnecessary assumptions, yet this forces our hand into making different, perhaps stronger assumption to get identification.

Please clear up my confusion in the comments if you are more in the know.

Posted by Matt Blackwell at 1:49 PM | Comments (5)

16 November 2009

Greiner on "Exit Polling and Racial Bloc Voting"

Please join us at the Applied Statistics workshop this Wednesday, November 18th at 12 noon when we will be happy to have Jim Greiner of the Harvard Law School presenting on "Exit Polling and Racial Bloc Voting: Combining Individual-Level and R x C Ecological Data." Jim has provided a companion paper with the following abstract:

Despite its shortcomings, cross-level or ecological inference remains a necessary part of many areas of quantitative inference, including in United States voting rights litigation. Ecological inference suffers from a lack of identification that, most agree, is best addressed by incorporating individual-level data into the model. In this paper, we test the limits of such an incorporation by attempting it in the context of drawing inferences about racial voting patterns using a combination of an exit poll and precinct-level ecological data; accurate information about racial voting patterns is needed to trigger voting rights laws that can determine the composition of United States legislative bodies. Specifically, we extend and study a hybrid model that addresses two-way tables of arbitrary dimension. We apply the hybrid model to an exit poll we administered in the City of Boston in 2008. Using the resulting data as well as simulation, we compare the performance of a pure ecological estimator, pure survey estimators using various sampling schemes, and our hybrid. We conclude that the hybrid estimator offers substantial benefits by enabling substantive inferences about voting patterns not practicably available without its use.

Both the paper and the technical appendix are on the course website.

The Applied Statistics workshop meets each Wednesday in room K-354, CGIS-Knafel (1737 Cambridge St). We start at 12 noon with a light lunch, with presentations beginning around 12:15 and we usually wrap up around 1:30 pm. We hope you can make it.

Posted by Matt Blackwell at 9:00 AM | Comments (0)

12 November 2009

Choosing variances in general linear models

Today I'm going to talk about a particular problem from my own research and will outline a method for choosing variances in general linear models (GLMs), but I am also asking a question.

The standard setup of GLMs is (roughly) the following. One hypothesizes that the conditional mean of the outcome variable (y), E[y|x], can be expressed as a function of a linear predictor x'b, or:

The function μ is referred to as the link function. Common choices for μ include both the identity and log link. One common question is why one would choose to use a GLM with, for example, a log link instead of estimating via OLS the regression model:

There are two principle objections to the OLS method. First, in the presence of heteroskedasticity it is difficult to transform predicted values of ln(y) into predicted values of y, although it is possible. Second, the OLS method throws out any data coming from observations with y=0.

Unfortunately, the choice of a link function is comparatively easy (in my view) compared with the next step of choosing an appropriate function for the variance of y given x, which must be prespecified in most GLMs*. In my work I have focused on choosing variance functions that are proportional to some power of the variance:

The trick, then, is to choose the correct power with various powers of the mean corresponding to Poisson (k=1), Gamma (k=2), and Wald (k=3), for example. In health econometrics this can be accomplished by using a modified Park test (due to Manning and Mullahy). In this procedure one first computes tentative parameter estimates for a GLM based on one's prior beliefs about the appropriate variance function (I typically use Gamma-like regressions for this). The linear predictors from the tentative regression can be used to get raw-scale residuals by applying the inverse link function. The modified Park test is to then regress the squared raw-scale residuals on a constant and the linear predictor in a GLM with a log link and the coefficient on the linear predictor then indicates which variance structure is most appropriate.

Now for the question. In health utilization data one often has data with a large number of zeros, for example, less than 10% of my sample uses mental health services in any given year. While GLMs are typically well behaved, in the presence of so many zeros this need not be the case. One common practice is to then use a "two part" model in which one uses an initial probit or logit regression to estimate the probability of any utilization and then estimate the second stage GLM model among users only. My question relates to the appropriate sample to use for the modified Park test--users or everybody? It turns that in this case it matters since when I look at everyone I get evidence in support of Gamma-like regressions (i.e. k=2 in my Park test), but when I only consider users in the Park test I get estimates of k=2.6, or so, which is more consistent with Wald-type variances.

My strong suspicion is that the latter approach is more appropriate since the GLM is only estimated among users, but I've hunted in the literature and found no specific advice on this point and many examples that seem to indicate that the test should be done on everybody.

* One exception is the Extended Estimating Equations method proposed by Basu and Rathouz (implemented as pglm in Stata).

Posted by Martin Andersen at 9:17 PM | Comments (4)

Bill Support by Page Length

There was a lot of press on the 1,000+-page length of the House health care bill, H.R. 3962. That got me thinking... didn't we hear the same thing about the stimulus bill and the Patriot Act? Aren't most "controversial" bills also very long?

It would make sense. Controversial bills require a lot more ink -- pork, special cases, exceptions -- to reel in support. Uncontroversial bills can be written succinctly and pass as is.

To assess this I scraped bills from OpenCongress, which maintains the full text, voting results and amendment history of House and Senate Resolutions. You can even comment on specific portions of bills. There's already a bunch of neat comments on potential loopholes in H.R. 3962.

I downloaded the text and voting results for all 152 House resolutions passed by the 111th House. A boxplot of page length against support appears below. Each page length group represents roughly 20% of House resolutions. The plot shows the suspected trend, that longer bills have less support. One-page bills almost always pass unanimously!

Posted by Kevin Bartz at 10:12 AM | Comments (1)

11 November 2009

Answering "why" questions

Brandon Stewart pointed me to an interesting blog post by Andrew Gelman that touches on the issue of explaining the "causes of effects." The basic point is that "why" questions are difficult to answer in a potential outcomes framework but often we really care about them. Some folks in political science have gone so far as to argue that researchers using "qualitative" methods are more inclined (and better able) to tackle these "why" questions than their "quantitative" colleagues who mostly focus on "effects of causes."

This has been on my mind lately -- as part of a class in the statistics department, I've had several conversations with Don Rubin about how retrospective "case-control" studies might fit into the potential outcomes framework. The goal of the medical researchers that execute these studies is usually a "why" question: why did an outbreak of rare disease X occur, which genes might cause breast cancer, etc. Case-control studies and their variants are great for searching over a number of possible causes and pulling out the ones that have strong associations with the outcome, but they aren't so great for estimating treatment effects. Rubin suggests that the proper way to proceed is probably to first use a case-control study to search over a number of possible causes and then estimate treatment effects for the most likely causes using a different sampling method (matched sampling for situations where the research has to be observational, experimentation when it's possible). It seems like this already happens to some extent in biostatistics and epidemiology and it also happens informally in political science.

I think this formulation suggests that answering a "why" question requires both "causes of effects" and "effects of causes" approaches; we need to search over a number of possible causes to identify likely causes, but we also need to test the effectiveness of each likely cause before we can say much about the causal effect. We probably still can't answer questions like "what caused World War I" but maybe this gets us somewhere with more tractable types of "why" questions.

Posted by Richard Nielsen at 4:30 PM | Comments (1)

7 November 2009

Just in time for "Superfreakonomics"

A friend recently pointed me to a 2007 New Republic article in which the author, Noam Scheiber, argues that the "Freakonomics" phenomenon is lamentable because it represents a trend toward research in which clever identification strategies are prized over attempts to answer what Scheiber calls "truly deep questions." Although two years and the publication date of a second Levitt and Dubner book have since passed, the article caught my attention because I have been considering a related issue of late. We are all well aware of how difficult it is to make causal inferences in the social sciences, so it is not surprising that researchers are drawn to settings in which some source of exogenous variation allows for identification of the influence of a specific causal factor. In fact, progress on those "truly deep questions" depends in part on this type of work. However, focus on clean identification has some potentially negative implications. Scheiber names one: answering questions of peripheral interest. A second, which is of greater concern for me, is concentrating on population subgroups that may or may not be of scientific interest in and of themselves and that, in either case, are unable to provide direct insights into broader population dynamics.

Thanks to Imbens and Angrist, we know that even when it is not possible to identify the population average effect of a "treatment" (i.e., causal factor of interest) on a given outcome, it is often possible to identify a "local average treatment effect," that is, the average effect of a treatment for the subpopulation whose treatment status is affected by changes in the exogenous regressor. This subpopulation is composed of so-called "compliers," who will take the treatment when assigned to take it and will not when they are not. Sometimes this subpopulation is of scientific or policy interest (for example, we may be interested in knowing the effect of additional schooling on earnings for those students who might drop out of high school but for compulsory education laws). Oftentimes, it is not. In contrast, the broader population and the portion of the population that receives treatment are almost always of interest. These groups are certainly policy-relevant (it would be misleading to project the effect of a drug on public health based only on the drug's effect amongst those who were induced to take the drug) and they are needed to generate "stylized facts" that help us organize our understanding of the social world. (Also, these groups can be observed whereas compliers are not a generally identified subpopulation.)

Unfortunately, when treatment effects are heterogeneous, the identified local average effect does not provide direct information about the wider population. This is problematic since treatment effects are likely to be heterogeneous in social science applications. In fact, this heterogeneity is one of the reasons why identifying causal effects is so difficult (individuals' self-selection into a treatment status based in part on anticipated treatment effects induces endogeneity problems).

A number of demographers have discussed the problem of extrapolating local average treatment effect estimates to the broader population. Greg Duncan, in his presidential address to the Population Association of America, stated that although causal inference is "often facilitated by eschewing full population representation in favor of an examination of an exceedingly small but strategically selected portion of a general population with the 'right kind' of variation in the key independent variable of interest.... a population-based understanding of causal effects should be our principal goal." Robert Moffitt writes that although "some type of implicit weighting is needed" to help us understand how to trade off internal and external validity, "this problem has not really been addressed in the applied research community." Some researchers have suggested using bounds for average treatment effects that are not point-identified (for example, Manski). Of course, the usefulness of bounding techniques depends on the tightness of the bounds, which in turn depends on what assumptions we are willing to impose - and it is exactly scholars' discomfort with prevailing assumptions (e.g., lack of correlation between the error and the treatment indicator) that drove the current focus on non-representative population subgroups. It seems to me that there is still work to be done to connect subpopulation causal estimates to broader population trends. I would be interested to hear of work in this area that you think is promising.

Posted by Deirdre Bloome at 8:02 PM | Comments (4)

3 November 2009

Airoldi on "A statistical perspective on complex networks"

I hope you can join us at the Applied Statistics Workshop this Wednesday, November 4th, when we will be happy to have Edo Airoldi, Assistant Professor in the Department of Statistics here at Harvard. Edo will be presenting a talk entitled "A statistical perspective on complex networks" for which he has provided the following abstract:

Networks are ubiquitous in science and have become a focal point for discussion in everyday life. Formal statistical models for the analysis of network data have emerged as a major topic of interest in diverse areas of science, as many scientific inquiries involve collections of measurements on pairs of objects. Probability models on graphs date back to 1959. Along with empirical studies in social psychology and sociology from the 1960s, these early works generated an active network community and a substantial literature in the 1970s. This effort moved into the statistical literature in the late 1970s and 1980s, and the past decade has seen a burgeoning network literature in statistical physics and computer science. The growth of the World Wide Web and the emergence of online networking communities such as Facebook and LinkedIn, and a host of more specialized professional network communities has intensified interest in the study of networks and network data. In this talk, I will review a few ideas that are central to this burgeoning literature. I will emphasize formal model descriptions, and pay special attention to the interpretation of parameters and their estimation. I will conclude by describing open problems and challenges for machine learning and statistics.

The Applied Statistics workshop meets each Wednesday in room K-354, CGIS-Knafel (1737 Cambridge St). We start at 12 noon with a light lunch, with presentations beginning around 12:15 and we usually wrap up around 1:30 pm. We hope you can make it.

Posted by Matt Blackwell at 10:47 AM | Comments (3)