2. Upon receiving a submission, the PPR Chair and Advisory Committee will select reviewers based on relevant expertise, stature in the field, and quality of prior PPRs.
3. The review process combines journal-style reviews (albeit more complete and constructive, see below) and, simultaneously, empirical replication of the papers.
4. We hope to produce outside reviews within about two weeks from when we get the information we need. More details on this below.
5. Authors retain full control over what they put in their papers, whether or not to make changes in response to anything or everything in their peer pre-reviews, and whether to pass this information on to journals (see below). Authors are required to acknowledge receiving assistance from the “Alexander and Diviya Maguro Peer Pre-Review Program at Harvard’s Institute for Quantitative Social Science”.
1. We ask peer pre-reviewers to write the same type of review they would for a scholarly journal, only much faster, more complete, and constructive (see here). Reviewers for academic journals often hunt for the first problem that may be a fatal flaw, which enables them to provide a recommendation to the journal and stop reading. Our reviewers are expected to read the entire paper and make constructive suggestions throughout.
2. Preparing a peer review for a scholarly journal rarely takes more than a day. Most of the 3-6 month wait for authors is the time it takes for the journal to find the reviewer and for the reviewers to begin work. In order to quickly launch the program, we offer reviewers financial compensation to accept the assignment and complete it faster. As such, speed should not reduce review quality. As the program grows, we plan to experiment with other types of incentives so that we can scale the program beyond IQSS. We have a detailed evaluation plan in place so that reviews get better and are completed faster over time.
4. At least at first, we are providing more than one reviewer for each paper, but not simultaneously as with a journal. Instead, we do them sequentially, with author revision prior to sending out the second round. (We may also experiment with doing all three reviews simultaneously, which would produce a better collection of reviews after one week but may be an inefficient use of our funds if the author made a mistake that used up all 3 reviews correcting the same mistake.)
5. IQSS internal staff, sometimes with help from graduate students not trained by the IQSS faculty author, complete an empirical replication of the draft paper. Staff will rerun all the analyses in the paper, dissect the computer code, evaluate the methodology, consider alternative statistical assumptions, and make suggestions. We (and journals that verify replicability of submitted datasets) have found that it is helpful when empirical replication processes are not anonymous, which allows the author and our staff to go back and forth to clarify analysis details.
6. The results of these replications will be made available to reviewers, who may also ask the replication staff to conduct further analysis (e.g. using a different set of statistical assumptions to test the robustness of the results).
7. Reviews are double blind in that authors and reviewers are unaware of each other’s identities (unless each chooses otherwise). The usual confidentiality agreements bind reviewers from using or revealing information in paper drafts.
8. Detailed metrics will be kept and evaluations conducted on the effectiveness of the PPR program; all aspects of the program are subject to change as we learn more. Authors must consent to their information being used (in confidential ways) for improving the PPR process.
1. If the author chooses and all reviewers consent, IQSS will send a package of information to the editor of a journal of the author’s choice about the peer pre-review process. The package will include all versions of the paper, all peer pre-reviews, date and time stamps, and (although not known by the authors before or after this) the identities of the reviewers. The author may also submit a memo, after the peer pre-review process, that details changes in the paper in response to the last set of peer pre-reviews (as is often expected for revise and resubmit rounds at journals). Editors may use this information as they see fit including sending the paper through their usual peer review process, having an abbreviated peer review process, or skipping to a decision immediately. Editors must agree to standard confidentiality rules before receiving PPR materials.
2. Several journal editors tell us that, with a package of papers, PPR reviews, and responses from the author, they will often speed up publication decisions and in some cases even skip the journal peer review process entirely.
3. As we obtain concrete agreements with specific journals, we will publish more information here.