The peer pre-review (PPR) program at Harvard’s Institute for Quantitative Social Science (IQSS) is an experiment designed to solve problems in science, and science journalism, by improving scholarship before it becomes public, speeding scientific discovery and publication, and reducing substantial inefficiencies for individual researchers. PPR, which is available for now to IQSS faculty affiliates, attempts to provide the following two services for draft papers prior to them becoming public: (1) a detailed, constructive, anonymous peer review by a leading outside scholar incented to participate, with extremely fast turnaround (probably a few weeks), and (2) a full empirical replication by our internal team. While we experiment and refine the program, we are making this service available for now to IQSS faculty affiliates; we are presently working out ways of including social scientists at other universities and research organizations.

Our experience indicates that papers that go through this process substantially increase in quality and and reduce the chance of miscommunications that lead to delayed peer review.  Results so far suggest that peer review is faster than it would be otherwise as well.

A generous gift from Alexander and Diviya Magaro makes this program possible.

Problems PPR Address

The Public Problem: After researchers draft academic papers, often years before publication in a scholarly journal, they post them on their homepage or as part of a working paper web repository. Reporters then read these drafts and publish news articles about them.  Unfortunately, the probability of a news article appearing about a paper depends more on how sensational the claims are than their veracity. Incorrect claims in academic preliminary paper drafts -- which is to be expected as a natural part of all science -- then can mislead journalists, the public, and other academics, as papers can stay on the web in unpublished form for years before authors learn of the problem.

Trying to “fix” journalism or train journalists will not solve this problem. Journalists already work hard trying to weed out papers with mistakes and or ones which other academics they interview think are flawed. The problem is that figuring out whether a claim is correct is an extremely difficult scholarly task requiring a community of scholars. We know that one scientist following all the rules of science cannot accomplish the task on their own, and journalists and their editors are obviously not trained as scientists. Indeed, the only solution ever devised that is at all reliable is the scientific community -- which works together in competition and cooperation in pursuit of the same goals, and with incentive structures that apportion credit more for moving the scholarly consensus further from the status quo, and proportionately increasing the evidence necessary for more surprising claims. The process works because it is easier to fool yourself than it is to fool others, and so the community is required to evaluate claims. Getting scientific claims right requires many people working on a problem together, checking and building on each other’s work.

The result is that a scholarly paper that makes it through the peer review process, is approved by the editor, and gets published in a leading scholarly journal is perhaps only slightly more likely to be correct than serious papers that remain unpublished. Yet, although the advantage to individual articles may be slight, the aggregate result of repeated application of the process has produced the majority of the technological and scientific knowledge the world has known for the last 400 years. Thus, instead of trying to teach individual scholars and journalists the wrong lesson by asking them to try to individually stand in for the entire scientific community, PPR will speed up the process of scientific discovery, in particular the time between the appearance of a draft of a paper and a scholarly publication, and will surface many problems with papers before papers appear publicly.   

The Scholar’s Problem: In the social sciences, when researchers finish paper drafts, they typically make them available to a few colleagues or trusted friends for comments, revise, and then post papers on their websites. Sometimes they also give papers at a conference (the hurdle for which in the social sciences is usually just a title, abstract, and a promise of a paper at a future date, and so not remotely the same hurdle as a scholarly publication), and they then revise further on the basis of comments received. Scholars then send their papers to journals for peer review. This first round of peer review almost always takes at least 3 months, and can take as long as 6 months or more.  Few papers are accepted on the first round, and so there’s some more revision, and then there is (if given a “revise and resubmit”) resubmission to the original journal or (if rejected) the process begins anew at a different journal. Then comes another 3-6 month waiting period. Papers from the best scholars commonly go through multiple rounds of review, during which they spend most of the time waiting (and possibly at reputational risk if their paper is public and wrong), and journalists and the public spend their time either with no access at all or with a paper that may be misleading or could be improved by the peer review process.

For academics, the peer review process is humbling and frustrating. It makes papers better, at least on average. It also takes what seems like forever. Authors often think they are misunderstood by reviewers (because they are!) but, if the reviewer missed a crucial point in a paper, the fault is the author’s for not putting the point in a place so that a semi-attentive reviewer (like a semi-attentive journal reader) could not miss it, and so the paper gets better. This whole process is one of the most difficult parts of being a scientist, but it is also one of the most important.