Social Science Statistics Blog: Statistics and the Death Penalty

May 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30	31

This blog makes public the hallway conversations about social science statistical methods and analysis from the Institute for Quantitative Social Science and related research groups. Expect to see posts on trends in methodological thought, questions and comments, paper and conference announcements, applied problems needing methodological solutions, and methodological techniques seeking applied problems. Also included are summaries of papers and comments from a popular weekly research workshop held here and billed as a tour of Harvard's statistical innovations and applications with weekly stops in different disciplines. [read more...]

« Gosnell Prize Winner | Main | SPM Career Achievement Award »

13 June 2007

Statistics and the Death Penalty

A few days ago, the AP moved a story reporting on academic studies of the deterrent effect of the death penalty on potential murderers. Many media outlets picked up the story under headlines such as "Studies say death penalty deters crime", "Death penalty works: studies", and my favorite, "Do more executions mean fewer murders?" Presumably the answer to the last question is yes, at least in the limit; if the state were to execute everyone (except the executioner, of course), clearly there would be fewer murderers.

I was surprised when I read the article on Monday morning, since my sense of the state of play in this area is that it is probably impossible to tell one way or the other. Those are the findings of a recent study by Donohue and Wolfers, which finds most existing studies to be flawed and, more importantly, points out a variety of reasons why estimating the correct deterrent effect is difficult in principle. Here is some of what Andrew Gelman had to say about their study last year:

My first comment is that death-penalty deterrence is a difficult topic to study. The treatment is observational, the data and the effect itself are aggregate, and changes in death-penalty policies are associated with other policy changes.... Much of the discussion of the deterrence studies reminds me of a little-known statistical principle, which is that statisticians (or, more generally, data analysts) look best when they are studying large, clear effects. This is a messy problem, and nobody is going to come out of it looking so great.
My second comment is that a quick analysis of the data, at least since 1960, will find that homicide rates went up when the death penalty went away, and then homicide rates declined when the death penalty was re-instituted (see Figure 1 of the Donohue and Wolfers paper), and similar patterns have happened within states. So it's not a surprise that regression analyses have found a deterrent effect. But, as noted, the difficulties arise because of the observational nature of the treatment, and the fact that other policies are changed along with the death penalty. There are also various technical issues that arise, which Donohue and Wolfers discussed.

Given the tone of the article (and certainly the headlines), you would have thought that the Donohue and Wolfers paper had been overlooked by the reporter, but no: he cites it in the article, and he interviewed Justin Wolfers! He seems to have missed the point, however; the issue is not that some studies say that "there is a deterrent effect" and some say "we're just not sure yet". The problem is that we aren't sure, and we probably never will be unless someone gets to randomly assign death penalty policy to states or countries. This raises a problem that we often face in social science: there are questions that are interesting, and there are questions that we can answer, and the intersection of those two categories is probably a lot smaller than any of us would like. This doesn't seem to be a realization that has crept into the media as of yet, so it is no surprise that studies that purport to give answers to interesting questions will get more coverage than those pointing out why those answers probably don't mean very much.

Posted by Mike Kellermann at June 13, 2007 4:19 PM

Comments

Donohue and Wolfers have been strongly rebutted.

(2006) "This analysis shows that attempts to make the deterrence effect disappear are ineffective." (p 16)
--- Existence of the death penalty, in law, has a statistically significant impact on reducing murders. (p 23)
--- Execution rates show significant impact in reducing murders. (p 13 & 23)
--- Death row commutations, and other removals, increase murders. (p13 & 23)
--- The criticism of our studies is flawed and does not effect the strength of the measured deterrent effect.
"The Impact of Incentives On Human Behavior: Can we Make It Disappear? The Case of the Death Penalty", Naci H. Mocan, R. Kaj Grittings, NBER Working Paper, 10/06, www(dot)nber.org/papers/w12631

(2006) " . . . (Donohue and Wolfers' "D&W") criticisms of Zimmerman's analysis are misrepresentative, moot or unsupportable in terms of the analyses they perform." "It is shown that Zimmerman's published empirical results, or the conclusions drawn from them, are not in any way refuted by D&W's critique." (pg 3) "This later estimate suggests that each execution deters 14 murders on average . . .". (pg 7) "It is shown that D&W made a number of serious misinterpretations in their review of Zimmerman's study and that none of the analyses put forward by D&W (which ostensibly refute Zimmerman's original results and conclusions) hold up under scrutiny. (pg8) " . . . D&W do not even report Zimmerman's "preferred" results correctly, and then proceed by carrying on this error throughout the remainder of their critique."(pg8) "Of course, (D&W's) omission tends to create a strong impression that Zimmerman's analysis 'purports to find reliable relationships between executions and homicides', when his actual conclusions regarding the deterrent effect of capital punishment are far more agnostic." (pg10) " . . . D&W's method of interpreting their results is not consistent with that proscribed by the received econometric literature on randomized testing . . .". "As such, D&W's interpretation of their randomized test in itself does not (and cannot) reasonably lead one to conclude that Zimmerman's estimates suggesting a deterrent effect of capital punishment are spurious." (pg12) " . . . D&W do not appear to have interpreted their randomization test in any meaningful fashion." (pg14) " . . . the state clustering correction employed by D&W may not be producing statistically meaningful results." (pg16) "And while D&W once lamented that recent econometric studies purporting to demonstrate a deterrent effect of capital punishment yield 'heat rather than light', as shown herein, their criticisms of Zimmerman (2004) tend to yield 'smoke rather than fire'."(pg26)
Zimmerman, Paul R., "On the Uses and 'Abuses' of Empirical Evidence in
the Death Penalty Debate" (November 2006). ssrn(dot)com/abstract=948424

Posted by: Dudley Sharp at June 13, 2007 10:01 PM

"[T]heir criticisms of Zimmerman (2004) tend to yield 'smoke rather than fire". Nice. Unfortunately, flowery rhetoric doth not a good statistical argument make. All I get with the link provided is an abstract, with a lot of flowery rhetoric, which, you guessed it, doth not impress me much.

The problem with this whole question is that it quickly becomes dominated by clowns eager to spout flowery rhetoric in advance of some deeply-held ideological belief which they think they can dress up in some obscure means of clustering their standard errors. That doesn't get us anywhere. So, let's deissuify this a bit and get back to the core statistical problem . . .

. . . which is that cross-sectional, regression-based studies of the social and/or economic impacts of policy changes are confined to the statistical garbage heap. You can NEVER solve the omitted variable problem. You can NEVER separate your conclusions from the functional form assumptions embedded in the analysis. At the end of the day, you can cluster your standard errors however you damn well want, but you are still just polishing a turd. Why do you think no credible economist runs growth regressions anymore? Because every credible economist now accepts that they are a waste of time. Yet some silly people apparently think they can conveniently forget all that social science has learned over the past 15 years just because they have seized upon an issue they think will get them some face time on the local news . . . we're here to stop that . . .

So, stop trying to get your name in the paper and start doing serious research. Quack, quack.

You say, "Donohue and Wolfers have been strongly rebutted." I saw, "Donohue and Wolfers have been strongly rebutted [by those who either have no idea what they are talking about or find it advantageous to have no idea what they are talking about in this instance]."

Posted by: The Randomizer at June 14, 2007 1:13 AM

Dear Randomizer:

The rebuttal was not from me, but from the authors of those studies.

You can look up the CVs of Zimmerman, Naci H. Mocan and R. Kaj Grittings, if you care to.

They're a pretty credible lot.

Posted by: Dudley Sharp at June 14, 2007 11:10 AM

Social Science Statistics Blog

Authors' Committee

Chair:

Members:

Weekly Research Workshop Sponsors

Recent Comments

Recent Entries

Categories

Blogroll

Archives

Notification

Statistics and the Death Penalty

Comments

Notification

Post a comment