rulerWe are committed to tracking the performance and impact of the Alexander and Divya Maguro Peer Pre-Review Program through a rigorous and continuous evaluation program.  The ultimate, long-term outcomes we hope to impact are time to publication, quality of scientific research, and scholarly productivity. We will do this by improving time-to-publication and research quality for our authors, and by developing systems and approaches which can be adopted more broadly in the scientific community.

Our initial evaluation plan focuses on measuring our success in improving the research process for participating scholars and participating reviewers.  We will also track the outcomes of all papers.  Eventually, as the program grows and scales, we will anticipate being able to perform meaningful comparisons to equivalent papers that have not been part of the PPR  program.

Authors are asked to complete short baseline evaluation survey, and will be asked to complete short follow-up surveys after receiving each review.  Authors will be asked to report to us when their paper is published, and will be asked to complete a final evaluation survey at that time.

Metrics tracked will include:

Review-Level Metrics:

Measured from Author Feedback:

  • Overall satisfaction with review
  • Satisfaction with quality of review
  • Satisfaction with timeliness of review
  • Satisfaction with thoroughness of review
  • Satisfaction with actionability of review
  • Number and nature of changes made to paper based on PPR’s
  • Perceived impact of changes on quality of paper

Measured from Reviewer Feedback:

  • Satisfaction with process
  • Satisfaction with appropriateness of paper given expertise
  • Satisfaction with clarity of review instructions
  • Satisfaction with communications
  • Satisfaction with amount of honorarium
  • Satisfaction with timeliness of payment of honorarium

Measured from Editors (for journal editors that agree to consider Peer Pre-Reviews)

  • Satisfaction with quality of reviews
  • Satisfaction with relevance of reviews
  • Satisfaction with thoroughness of reviews
  • Willingness to expedite review process for peer pre-review papers
  • Willingness to substitute peer pre-reviews for regular journal reviews

Paper-Level Metrics:

  • Time to submission (first journal attempt)
  • Time to submission relative to author expectation at program start
  • Quality of journal relative to author expectation at program start
  •  Number of unsuccessful submissions prior to acceptance
  • Number of journal-based desk-rejections
  • Number of journal-based rejections with full reviews
  • Whether the paper is successfully published
  • Quality of journal of eventual publication (impact-factor)
  • Quality of journal (impact-factor) relative to author expectation at program start
  • Time from submission to acceptance relative to journal average
  • Time from submission to publication relative to journal average
  • Number of journal-sponsored reviews for paper
  • Level of required revisions prior to publication (major, minor, none)
  • Time to publication
  • Time to publication relative to author expectation at program start