Social Science Statistics Blog

November 17, 2005

Social Science and Litigation, Part IV

Jim Greiner

In previous blog entries here, here, and here, I discussed the fundamental questions about the objectivity of expert witnesses raised by Professor of History Morgan Kousser's article entitled "Are Expert Witnesses Whores?".

In my view, Professor Kousser's article suggests that expert witnesses are not fully aware of the threat to their objectivity that the litigation poses. For example, despite acknowledging that lawyers "peform[ed] most of the culling of primary sources" in the cases in which he offered testimony, Professor Kousser argues, for a number of reasons, that there was no threat to objectivity. Primary among these reasons was the adversarial process, which gave the other side an incentive to find adverse evidence and arguments, and thus an incentive for an expert's own attorneys to share such evidence and arguments.

Professors Kousser's reasoning dovetails with private conversations I've had with social scientists about litigation experiences, who also insisted that they retained their objectivity throughout. Invariably, they support this contention by describing critical moments during pre-trial preparation in which they refused requests from their attorneys to testify to something, saying that the requests pushed the data too far or contradicted their beliefs.

My response: think about what the attorneys had already done to your objectivity before you reached these critical moments. Might they even have pushed you into refusing so as to convince you of your own virtue?

Professor Kousser and other social scientists have misperceived the nature of the threat. Professor Kousser is correct when he suggests that lawyers, upon encountering a potentially damaging piece of source material or evidence within an expert's area, are unlikely to suppress it (in the hope that the other side is negligent). But we lawyers do accompany our transmission of the potentially damaging item with rhetoric about its lack of reliability, importance, or relevance. Similarly, when we prepare experts for deposition and trial, we do not avoid adverse arguments or potential weaknesses in reasoning. Instead, we raise them in a way so as to minimize their impact. Often, we (casually) use carefully tailored, ready-made rhetorical phrases about the issue, hoping to hear those phrases again at trial. Before conducting pretrial meetings with important experts, we meet amongst ourselves to decide how best to ask questions and discuss issues to "prop up" expert' resolve.

Social scientists have long known that the way a questioner phrases an inquiry affects the answer received, that the way in which a conversational subject is raised affects the opinions discussants will form. Perhaps social scientists believe that their knowledge of these phenomena makes them immune to such effects. My experience in prepping social scientist expert witnesses suggests that such is not the case.

Posted by SSS Coauthors at 2:54 AM

November 9, 2005

Social Science and Litigation, Part III

Jim Greiner

Continuing with the theme of quantitative social science expert witnesses in litigation introduced here and here, I shift gears to consider the experts' view of lawyers. Several expert witnesses with whom I have spoken confided that they often form low opinions of the lawyers who retain them. One common complaint is that the attorneys do not take the time to understand the guts of the issue experts were hired to examine. Another is that lawyers are uncommunicative and provide poor guidance as to their preferences for the testimony of experts.

Without question, poor lawyering is common, and some of what experts experience can be safely attributed to this source. But as was the case with lawyers' complaints about experts, experts' complaints about lawyers have their genesis partly in the structural rules that govern litigation. In most courts and jurisdictions, communications between a testifying expert and any other participant in the case (lawyer, fact witness, another expert) are discoverable. That means that, before trial, the other side is entitled to request, for example, copies of all email communications between lawyer and expert. In deposition, an expert may be questioned on telephone and other oral conversations with the retaining attorney. For this reason, good lawyers are careful about what they say to experts; they know that written or transcribed communications reach both parties to a case.

As is usually the situation, there are good reasons for this rule. An expert witness is one of the most dangerous creatures to enter a courtroom. By definition, he or she invariably knows more about the subject matter of the testimony than anyone else involved in the litigation, except perhaps the opposing expert. The judge and jury lack the knowledge and training to assess what the expert says. Thus, the law provides that experts must disclose anything that might form the basis of an expert's opinion, including communications with trial counsel (along with workpapers, references consulted, and other items).

Expert witness frustration aside, this discovery rule has other negative side effects; it affects not only how well lawyers prepare a case for trial, but also the treatment of the suit more generally. Parties and their attorneys need information to settle, and a lack of clear communication between lawyer and expert may cause the former to misjudge the settlement value of a case. Once again, we see how atypical Professor Kousser's experience as an expert was (see here), as lawsuits concerning the internal structure of a municipality or a state entity settle less often than, say, employment discrimination class actions.

In closing, a word to potential and actual social science expert witnesses: If you find yourself frustrated by a certain reticence or irrational exuberance on the part of the attorney retaining you, remember, there may be good reason for it.

Posted by SSS Coauthors at 2:50 AM

November 4, 2005

Measuring Social Trust

You, Jong-Sung

I had a very embarrassing experience, when I presented my early draft paper on “Inequality and Corruption as Correlates of Social Trust� at a Work-in-Progress Seminar at the Kennedy School of Government last fall. Professor Edward Glaeser came to my talk, but I was not aware of him although I had read his articles including one about “measuring trust." He asked a question about measurement of social trust without identifying himself. Since I had already talked about the problem of measurement (apparently he did not hear that because he was late) and was about to present my results, I did not want to spend much time about the measurement issue. He was not satisfied with my brief answer and repeated his questions and comments, saying that the typical trust question in surveys, “Do you agree that most people can be trusted or you can’t be too careful?�, may reflect trustworthiness rather than trust “according to a study.� Because I assumed that trust and trustworthiness reinforce one other, I did not think that was a great problem.

Our encounter was an unhappy one for us both. Probably he had an impression that I did not respect him and did not give adequate attention and appreciation to his questions and comments, and I was also kind of annoyed by his repeated intervention. One thing that made the things even worse was that I am not a native English speaker; I have particular difficulty with husky voices like his, a difficulty made the interaction even more problematic. After the talk, I asked him to give the reference for the study on measurement of trust he mentioned. He wrote down Glaeser et al. (2000), and I realized that I had read the article he cited. Even then, I was unaware who he was. I asked a participant of the seminar who he was, and to my surprise, he was Edward Glaeser, the lead author of the article on measuring trust. If I had recognized him, I would have paid much more attention to his questions and comments and tried to answer them better. How big a mistake I made!

Although I still think that the typical trust question captures both trust and trustworthiness, Glaeser et al.’s experimental results may indicate the trust question needs to be designed better. One thing to note in this regard is that caution is not the opposite of trust, as Yamagishi et al. (1999) argued. In my case study of social trust in Korea, I found that inclusion and exclusion of “being careful� option in trust questions produced substantially different results. More respondents agree that most people can be trusted when they were simply asked, “Do you think most people can be trusted� than when they were given the two options “trusting most people� and “being careful.� Average percentage of trusting people was 42.9 per cent for the former type of questions, and 32.2 per cent for the latter type of questions. I looked at the GSS, and the same was true there. The trust question was given without the option of being careful once during 1983-87, and 55.7 per cent of respondents agreed that most people can be trusted. When the “being careful� option was given, only 42.1 per cent of respondents did so.

Posted by SSS Coauthors at 5:54 AM

October 24, 2005

“IV Etiquette�

You, Jong-Sung

One of my most embarrassing experiences occurred surrounding the use of instrumental variables in my ASR article with Sanjeev Khagram on inequality and corruption (2005). The article developed from my qualifying paper on causes of corruption (2003), in which I examined several hypotheses on the causal effects of inequality, democracy, economic development, and trade openness. Since all these four explanatory variables may be affected by corruption, I tried to find appropriate instruments. Initially, I tried five: latitude, # frost days, malaria prevalence index, ethno-linguistic fractionalization, and constructed openness. They had a strong predictive power for the endogenous variables in the first stage regression, and the p-values for the over-identification test in the second stage regressions were generally large enough so that I could not reject the null hypothesis of no correlation between the instruments and the error term of the regression. I worked with Professor Khagram to make a publishable article from my qualifying paper, and we submitted our manuscript to the ASR. The first review we received from the editor was encouraging. The editor advised us to “revise and resubmit� in his three-page long letter, which showed his interest in our paper. But the editor as well as an anonymous reviewer asked us to provide an argument explaining how our instruments were correlated with the endogenous variables but not directly correlated with corruption. I initially considered responding to this critique by citing Rodrik et al.’s draft paper entitled “Institutions Rule: The Primacy of Institutions over Geography and Integration in Economic Development� (later published in the Journal of Economic Growth, 2004), which argued, “An instrument is something that simply has some desirable statistical properties. It need not be a large part of the causal story.�

However, I was criticized regarding the use of instruments when I presented at a Work-in-Progress Seminar at the Kennedy School of Government and at Comparative Political Economy Conference at Yale University in spring 2004. In the Work-in-Progress Seminar, some professors at the Kennedy School noted that overidentification test can pass if they are all wrong in the same direction. In the Yale conference, Professor Daron Acemoglu of MIT was a discussant for my paper, and he used the term “IV etiquette� to emphasize the importance of giving a plausible story for the first stage. He pointed that without a clear story for the fist stage, it is impossible to tell whether the instrument is uncorrelated with unobserved determinants of the dependent variable. It was really an embarrassing moment when I was criticized for the lack of etiquette in front of many scholars.

So, I had to find more convincing instruments. In this regard, I have to thank my friend, Andrew Leigh, who was a doctoral student in public policy then and is currently Research Fellow at Australian National University. He found that “mature cohort size� can be used as an instrument for inequality in his dissertation paper entitled "Does Equality Lead to Fraternity?", based on Higgins and Williamson's (1999) theory of cohort size effect on income inequality. Also, I came to realize how conference presentations and discussions can be helpful in improving the quality of research.

Posted by SSS Coauthors at 4:04 AM

October 20, 2005

Book Review: “The Probability of God�, by Stephen D. Unwin

Drew Thomas

I continue with my review of The Probability of God, by Stephen D. Unwin, which I began here.

The first clue I had that this book would have anything but rigorous mathematical analysis was that I found it in the Harvard’s Divinity library. As expected, the book is mainly philosophical in nature, but that doesn’t mean it exceeds its mathematical scope. Indeed, it gives the reader a good introduction to Bayesian inference while being very clear about its limits.

The premise is simple: start with a proposition – in this case, that a monotheistic God exists; select a series of evidential questions that are relevant to the investigation; and assess the evidence under each of the two mutually exclusive probabilities.

The considerations he takes into account are as follows:

Prior distribution: Is there any reason to believe God exists other than using anti-anthropic arguments? Unwin believes there is no value in the “watchmaker� hypothesis – that the wonder and beauty we see around us is so complex that it could only have been designed by a being of higher order than our own – and so chooses the simplest of priors, that there might as well be a 50-50 chance. (Unwin later demonstrates that this prior fails any reasonable sensitivity analysis – stay tuned.)

In its rawest form, Bayesian inference takes the following form:

P(proptrue|evid) = P(evid|proptrue)P(proptrue)
-------------------------------------------------------------
P(evid|proptrue)P(proptrue) + P(evid|propfalse)P(propfalse)

Notice that if we divide top and bottom by P(evidence|prop false), we have the following quantity on top and bottom: P(evidence|proptrue)/P(evidence|propfalse). Statisticians call this a Bayes Factor – the likelihood of one model over another – while Unwin, seeking to appeal to a wider audience, calls this a Divine Indicator. I’ll continue with the former.

He then considers six “quantities� that relate to God’s existence, and how they fair under a world with God or no-God. In particular, he examines each Bayes factor, considers each piece of evidence to be independent from the others, then performs the Bayes calculation one at a time, using each subsequent posterior probability as the new prior probability. Any skeptic might question that the nature of his inquiries might be skewed under his own personal biases should remember that this is just an exercise.

In addition, to simplify the math, Unwin uses a scale of 1 to 3 to evaluate each piece of evidence, indicating no, weak or strong support (this is my interpretation, rather than a hard ranking system the author himself uses.) To put this into the equation, he uses a 5-level scale, setting the Bayes factor to be 0.1, 0.5, 1, 2 or 10 depending on the comparison of evidence.

1) The recognition that “goodness� exists. Under God, he argues, good and evil are built into the system. Without God, goodness can only be described as a pragmatic measure, so goodness wouldn’t be taken in that context. Unwin starts off with a blast and gives himself a 10. P(God exists) is now 91%.

2) The recognition that “moral evil� exists. Unwin says that moral evil is inevitable in a godless universe, but that God wouldn’t tolerate such a degree we have right now. Strong meets weak; the Bayes factor at this step is 0.5, leaving an 83% chance. (I find this step a little unsettling, as it immediately turns God into a humanlike figure, attaching too much specificity in my mind.)

3) The recognition that “natural evil� exists. In the wake of Hurricane Katrina, a great number of survivors in Louisiana are asking themselves what kind of a God would allow such a tragedy to happen. Unwin carries the same spirit across and claims that such a perspective makes little sense under God’s domain. No evidence versus strong gives a Bayes factor of 0.1 and a 33% chance of God’s existence.

4) The incidence of “intra-natural� miracles (such as whether praying for the Red Sox to win makes it so.) There are studies carried out routinely whether organized prayer can aid in the healing process. Never mind that these studies are highly unscientific – there isn’t an equal group praying against another injured person with roughly the same path to recovery, and a control group is nearly impossible to manufacture. Unwin doesn’t mind the inconclusiveness of these experiments; instead he relies on personal perspective and finds that prayer has some place in the world of God but little in one without. A Bayes
factor of 2 brings the probability of God back to 50%.

5) The incidence of “extra-natural� miracles (those examples that can’t be explained by science). These sorts of miracles were observed before God, so Unwin says many other systems are good enough to explain their existence (though certainly not their cause.) Equal evidence means a Bayes factor of 1, and the probability of God holds at 50%.

6) Religious experiences. I find this category to be the weakest of Unwin’s areas of evidence, since it immediately suggests a stacked deck. Unwin does hold back and merely suggests that what we perceive to be religious experiences – perceived moments of oneness with a higher power – are more likely to be justified if there is such a higher power. Unwin gives a Bayes factor of 2, bringing us to the conclusion that in his perspective, the probability of God’s existence is 67%.

Now many of you (including my co-authors) are bewildered as to why I’d consider this book, and this analysis, as being relevant to the practice of statistics. To begin with – or rather, end with – Unwin admits that this test is extremely sensitive to the choice of prior beliefs. Under his assessment of the evidence, his prior belief in God’s existence (50%) yields the probability of God’s existence at 67%; using prior beliefs of 10% or 75%, using the same evidence, swings the result to 18% or 86% respectively.

As in many strong works of philosophy, the important lesson is not in the answer, but in asking the questions that lead there. These calculations lead only to the halfway point of the text, as Unwin now segues from his method of observation into a discussion of the nature of faith, and what components of probability and faith lead to what we understand as belief.

Posted by SSS Coauthors at 5:31 AM

October 14, 2005

Instrumental Variables in Qualitative Research

You, Jong-Sung

In large-N quantitative research, instrumental variables are often used to address the problem of endogeneity. In small-N qualitative research such as comparative historical case studies, researchers examine historical sequence and intervening causal process between an independent variable(s) and the outcome of the dependent variable in order to establish causal direction and illuminate causal mechanisms (Rueschemeyer and Stephens 1997). However, careful examination of sequence and intervening process through process-tracing may not solve the problem of endogeneity. When Y affected X initially and X, in turn, influenced Y later, looking at the sequence and intervening causal process in the latter part without examining the former process will produce a misleading conclusion.

In my comparative historical case study of corruption in South Korea, relative to Taiwan and the Philippines, I attempted to test my hypothesis that income inequality increases corruption and to identify causal mechanisms. It was easy to show the correlations between inequality and corruption. Both inequality and corruption have been the highest in the Philippines and the lowest in Taiwan, with Korea in between. I found that the success of land reform in Korea and Taiwan produced much lower levels of inequality in assets and income than was true of Philippines, where land reform failed. I provided plausible evidence that the different levels of inequality due to success and failure of land reform accounted for different levels of corruption, and identified some causal mechanisms. Also, between Korea and Taiwan, I found that Korea's chaebol (large conglomerate)-centered industrialization and Taiwan's avoidance of economic concentration led to a divergence of inequality over time, which contributed to divergence of corruption level.

However, the process-tracing for the period after the success or failure of land reform and for the period after the adoption of different industrial policies was not sufficient to establish causal direction because different levels of corruption might have influenced the success and failure of land reform as well as the industrial policy. Hence, I had to show that success and failure of land reform was affected very little by corruption, but largely determined by external factors such as the threat of communism and the differences in the US policy toward these countries. Also, I had to provide evidence that the initial adoption of different industrial policies by Park Chung-hee in Korea and by the KMT leadership in China were not affected by the different levels of corruption. Essentially, land reform and industrial policy played the role of instrumental variables in statistical studies. These were exogenous events that produced different levels of inequality and thereby caused different levels of corruption but had not been influenced by corruption. Thus, the idea of instrumental variable can be useful in qualitative research as well.

Posted by SSS Coauthors at 3:29 AM

October 3, 2005

Multilevel Hazard Models in Log-Time Metric?

Felix Elwert

Has anybody figured out how to estimate multilevel hazard models with time-varying covariates in log-time metric (i.e., an accelerated failure time model)?

Together with two colleagues from the Medical School, I’m working on the effect of contextual variables on mortality. We're using a large longitudinal dataset of around ½ million married couples and nine years of follow up. Our key independent variable is time varying. In recent years, much work has been done on multilevel hazard models, for example, that done by Harvey Goldstein and colleages. But the standard recommendation for estimating such models in the presence of time-varying covariates is to approximate the Cox proportional hazard model using a conditional (i.e, fixed effects) logistic regression, which makes hefty demands on memory. Given the size of our data, we can implement this standard strategy only for a subset of our data.

We are hoping that the log-time metric would make better use of memory and allow us to use the entire sample. The question is: has anybody already developed software to estimate multilevel hazard models with time-varying covariates in the log-time metric? Or can't it be done in principle? Either way, I'd be grateful for pointers.

Posted by SSS Coauthors at 6:00 AM

September 27, 2005

Non-compliance, Bayesian Inference and School Vouchers: A Thesis Defense

Drew Thomas

Proceedings in the Harvard Dept. of Statistics seminar series started early this year, as Hui Jin eloquently delivered her doctoral thesis defense on Wednesday, September 14, entitled "Principal Stratification for Causal Inference with Extended Partial Compliance." Jin applied her ideas both to drug trials and to school choice (voucher) programs. She spoke in particular about the second application, focusing on a study of vouchers as offered to students from low-income families in the New York City public school system. In this study, 1000 students were offered a subsidy to help pay tuition for a private school of their choice, and were matched with students with similar conditions who were not offered the grant. Both groups were tracked for three years, and a set of tests at the beginning and end were used to measure achievement. The compliance factor was whether grant recipients would always take advantage of the offer, and whether unlucky ones would never make their own way to private school. While the compliance rate after three years remained high - roughly 80% - it was the compliance factor that proved to be the most instructive on the achievement pattern of students, a result found by stratifying the outcomes according to compliance patterns.

Those students expected to comply perfectly - attend private school with the grant and public school without it, in all three years - made the least improvement as compared to their colleagues in the other strata. Comparative performance improved with non-compliance; the biggest non-conformers, those who attended private or public school regardless of whether the grant was offered showed the most improvement over their previous scores.

Notably, the reasons for this performance haven't been completely explained, though Prof. Rubin (Jin's advisor and collaborator on the project) suggests that perhaps using the voucher as a threat to remove a student from his friends may compel a higher performance at public school. Whatever the underlying mechanism, the results give strong and compelling reason to fully consider the effect of vouchers in the school system.

Posted by SSS Coauthors at 7:00 AM

September 26, 2005

Applied Statistics - Xihong Lin

Mike Kellermann

This week's Applied Statistics Workshop presentation will be given by Professor Xihong Lin of the Department of Biostatistics at the Harvard School of Public Health. Professor Lin received her Ph.D. in Biostatistics from the University of Washington. She is one of the newest members of the Harvard statistical community, having just moved to Harvard from the University of Michigan School of Public Health. She has published widely in journals including the American Journal of Epidemiology, Biometrika, and the Journal of the American Statistical Association. She currently serves as the co-ordinating editor of Biometrics. Among her other awards, she has been recognized as an outstanding young scholar by both the American Statistical Association and the American Public Health Association.

Professor Lin's presentation, "Causal Inference in Hybrid Intervention Trials Involving Treatment Choice," considers the problem of causal inference from experiments in which some subjects are allowed to choose the treatment that they receive. Allowing treatment choice may increase compliance levels, but creates inferential challenges not present in a fully randomized experiment. Professor Lin will discuss her approach to this problem on Wednesday, September 28 at noon in Room N354, CGIS North, 1737 Cambridge St. Lunch will be provided.

Posted by SSS Coauthors at 11:53 AM

September 19, 2005

Conference on Methods in Health Services and Outcomes in Boston, October 28-30

Sebastian Bauhoff

Boston will host the 2005 International Conference on Health Policy Research from October 28-30. This year's theme is "Methodological Issues in Health Services and Outcomes Research" and presentations are meant to convey both content and methodology.

The conference includes a slightly eclectic selection of workshops on methods and the use of well-known health datasets -- two workshops on the latter are free, others cost $60 or $30 (students). Registration is not free either but studentes pay only $80. Looks interesting and useful overall, though you might want to attend selectively.

For more info check the conference website.

Posted by SSS Coauthors at 10:50 PM

September 18, 2005

Censoring Due to Death: A Statistics Symposium

Drew Thomas

The Harvard Dept. of Statistics kicks off its 2005-2006 seminar series on Monday, September 19 with a talk by the father of the Rubin Causal Model himself, Prof. Donald Rubin. An entertaining speaker if there ever was one, Prof. Rubin will give a firsthand account of his research to all who are interested.

The talk will be held in Science Center 705 at 4:00; a reception will follow. Looking forward to seeing all interested parties in attendance.

Posted by SSS Coauthors at 5:05 PM