Advertisement

Longitudinal Trends in the Performance of Scientific Peer Reviewers

      Study objective

      We characterize changes in review quality by individual peer reviewers over time.

      Methods

      Editors at a specialty journal in the top 11% of Institute of Scientific Information journals rated the quality of every review, using a validated 5-point quality score. Linear mixed-effect models were used to analyze rating changes over time, calculating within-reviewer trends plus predicted slope of change in score for each reviewer. Reviewers at this journal have been shown comparable to those at other journals.

      Results

      Reviews (14,808) were performed by 1,499 reviewers and rated by 84 editors during the 14-year study. Ninety-two percent of reviewers demonstrated very slow but steady deterioration in their scores (mean –0.04 points [–0.8%] per year). Rate of deterioration was unrelated to duration of reviewing but moderately correlated with mean reviewer quality score (R=0.52). The mean score of each reviewer's first 4 reviews predicted subsequent performance with a sensitivity of 75% and specificity of 47%. Scores of the group stayed constant over time despite deterioration because newly recruited reviewers initially had higher mean quality scores than their predecessors.

      Conclusion

      This study, one of few tracking expert performance longitudinally, demonstrates that most journal peer reviewers received lower quality scores for article assessment over the years. This could be due to deteriorating performance (caused by either cognitive changes or competing priorities) or, to a partial degree, escalating expectations; other explanations were ruled out. This makes monitoring reviewer quality even more crucial to maintain the mission of scientific journals.
      To read this article in full you will need to make a payment
      ACEP Member Login
      ACEP Members, full access to the journal is a member benefit. Use your society credentials to access all journal content and features.
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Purchase one-time access:

      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect

      References

        • International Commitee of Medical Journal Editors
        Uniform requirements for manuscripts submitted to biomedical journals: ethical considerations in the conduct and reporting of research: peer review.
        (Accessed December 29, 2009)
        • Baxt W.G.
        • Waeckerle J.F.
        • Berlin J.A.
        • et al.
        Who reviews the reviewers?.
        Ann Emerg Med. 1998; 32: 310-317
        • Schroter S.
        • Black N.
        • Evans S.
        • et al.
        What errors do peer reviewers detect, and does training improve their ability to detect them?.
        J R Soc Med. 2008; 101: 507-514
        • Callaham M.
        The evaluation and training of peer reviewers.
        in: Godlee F. Jefferson T. Peer Review in Health Sciences: Second Edition. 2nd ed. BMJ Books, London, England2003: 164-182
        • Callaham M.L.
        • Wears R.L.
        • Waeckerle J.F.
        Effect of attendance at a training session on peer reviewer quality and performance.
        Ann Emerg Med. 1998; 32: 318-322
        • Callaham M.L.
        • Schriger D.L.
        Effect of structured workshop training on subsequent performance of journal peer reviewers.
        Ann Emerg Med. 2002; 40: 323-328
        • Callaham M.L.
        • Knopp R.K.
        • Gallagher E.J.
        Effect of written feedback by editors on quality of reviews: two randomized trials.
        JAMA. 2002; 287: 2781-2783
        • ISI Web of Knowledge
        Journal citation reports 2008: ISI Web of Knowledge, Thomson Reuters; 2008.
        (Accessed February 10, 2010)
        • Callaham M.L.
        • Baxt W.G.
        • Waeckerle J.F.
        • et al.
        Reliability of editors' subjective quality ratings of peer reviews of manuscripts.
        JAMA. 1998; 280: 229-231
        • van Rooyen S.
        • Black N.
        • Godlee F.
        Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts.
        J Clin Epidemiol. 1999; 52: 625-629
        • Landkroon A.P.
        • Euser A.M.
        • Veeken H.
        • et al.
        Quality assessment of reviewers' reports using a simple instrument.
        Obstet Gynecol. 2006; 108: 979-985
        • Wessely S.
        Peer review of grant applications: what do we know?.
        Lancet. 1998; 352: 301-305
      1. Green SM, Callaham M. Implementation of a journal peer reviewer stratification system based on quality and reliability. Ann Emerg Med. In press.

        • Fitzmaurice G.M.
        • Laird N.M.
        • Ware J.H.
        Applied Longitudinal Analysis.
        Wiley-Interscience, Hoboken, NJ2004
        • Sullivan L.M.
        • Dukes K.A.
        • Losina E.
        Tutorial in biostatistics.
        Stat Med. 1999; 18: 855-888
        • Tercier J.
        • Callaham M.
        Qualitative profile of journal peer reviewers and predictors of peer reviewer quality.
        (Accessed February 10, 2010)
        • Schroter S.
        Effects of training on quality of peer review: randomised controlled trial.
        BMJ. 2004; 328: 673-678
        • Nylenna M.
        • Riis P.
        • Karlsson Y.
        Multiple blinded reviews of the same two manuscripts.
        JAMA. 1994; 272: 149-151
        • Rennie D.
        Editorial peer review: its development and rationale.
        in: Godlee F. Jefferson T. Peer Review in Health Sciences. 2nd ed. BMJ Books, London, England2004: 1-13
        • Wicherts J.
        • Dolan C.
        • Hessen D.
        • et al.
        Are intelligence tests measurement invariant over time?.
        Intelligence. 2004; 32: 509-537
        • Callaham M.L.
        • Tercier J.
        The relationship of previous training and experience of journal peer reviewers to subsequent review quality.
        Plos Med. 2007; 4 (Accessed February 10, 2010)
        • Stossel T.P.
        Reviewer status and review quality.
        N Engl J Med. 1985; 312: 658-659
        • Black N.
        • van Rooyen S.
        • Godlee F.
        • et al.
        What makes a good reviewer and a good review for a general medical journal?.
        JAMA. 1998; 280: 231-233
        • Evans A.T.
        • McNutt R.A.
        • Fletcher S.W.
        • et al.
        The characteristics of peer reviewers who produce good-quality reviews.
        J Gen Intern Med. 1993; 8: 422-428
        • Kliewer M.A.
        • Freed K.S.
        • DeLong D.M.
        • et al.
        Reviewing the reviewers: comparison of review quality and reviewer characteristics at the American Journal of Roentgenology.
        AJR Am J Roentgenol. 2005; 184: 1731-1735
        • Choudhry N.
        • Fletcher R.
        • Soumerai S.
        Systematic review: the relationship between clinical experience and quality of health care.
        Ann Intern Med. 2005; 142: 260-273
        • Ericsson K.A.
        Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains.
        Acad Med. 2004; 79: S70-S81
        • Ericsson K.
        An expert-performance perspective of research on medical expertise: the study of clinical performance.
        Med Educ. 2007; 41: 1124-1130
        • Eva K.
        The aging physician: changes in cognitive processing and their impact on medical practice.
        Acad Med. 2002; 77: S1-S6
        • Norcini J.
        • Kimball H.
        • Lipner R.
        Certification and specialization: do they matter in the outcome of acute myocardial infarction?.
        Acad Med. 2000; 75: 1193-1198
        • Krampe R.
        • Charness N.
        Aging and expertise.
        in: Ericsson K. Charness N. Feltovich P. The Cambridge Handbook of Expertise and Expert Performance. Cambridge University Press, New York, NY2006: 723-742
        • Ericsson K.
        • Lehmann A.
        Expert and exceptional performance: evidence of maximal adaptation to task constraints.
        Ann Rev Psychol. 1996; 47: 273-305
        • Kanazawa S.
        Why productivity fades with age: the crime-genius connection.
        J Res Pers. 2003; 37: 257-272
        • Levin S.G.
        • Stephan P.E.
        Research productivity over the life cycle: evidence for academic scientists.
        Am Econ Rev. 1991; 81: 114-132
        • Stephan P.
        • Levin S.
        Age and the Nobel Prize revisited.
        Scientometrics. 1993; 28: 387-399
        • Eva K.W.
        • Cunnington J.P.W.
        The difficulty with experience: does practice increase susceptibility to premature closure?.
        J Contin Educ Health Prof. 2006; 26: 192-198
        • Tercier J.
        • Callaham M.L.
        A normative model of peer review: qualitative assessment of manuscript reviewers' attitudes towards peer review.
        (Accessed February 10, 2010)
        • Bryson M.
        • Bereiter C.
        • Scardamalia M.
        • et al.
        Going beyond the problem as given: problem solving in expert and novice writers.
        in: Sternberg R.J. Frensch P.A. Complex Problem Solving: Principles and Mechanism. Lawrence Erlbaum, Hillsdale, NJ1991: 61-84
        • Shashok K.
        Content and communication: how can peer review provide helpful feedback about the writing?.
        BMC Med Res Methodol. 2008; 8 (Accessed February 10, 2010): 3https://doi.org/10.1186/1471-2288-8-3
        • Kruger J.
        • Dunning D.
        Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments.
        J Pers Soc Psychol. 1999; 77: 1-14
        • Eva K.
        • Cunnington J.
        • Reiter H.
        How can I know what I don't know?.
        Adv Health Sci Educ. 2004; 9: 211-224