The Science of Peer Reviewed Publication

It is well known that the peer review process for research has many flaws. Annals editors have for years been very active in performing research to better understand and improve the art of scientific publication; in fact, we are among the most active journals of any discipline on this issue. We believe that medical journals have a responsibility to perform rigorous self examination and share the results with the public. Not to do so is akin to to a physician treating patients with medications and therapies that have never been studied as to their efficacy and safety, and not bothering to monitor whether or not their patients are getting better.

Articles published in Annals

Articles by Annals editors in other journals

Effect of revealing authors’ conflicts of interests in peer review: randomized controlled trial
Leslie K John, George Loewenstein, Andrew Marder, Michael L Callaham
BMJ 2019; 367 doi: (Published 06 November 2019)

Declaring interests and restoring trust in medicine
Carl Heneghan, Margaret McCartney
BMJ 2019; 367 doi: (Published 06 November 2019)

Industry payments to physician journal editors
Victoria S. S. Wong, Lauro Nathaniel Avalos, Michael L. Callaham
PLoS One. 2019 Feb 7;14(2):e0211495. doi: 10.1371/journal.pone.0211495.

Is the relationship among outcome variables shown in randomized trials?
Schriger DL, Cooper RJ, Lopez-O'Sullivan A, Wystrach C, Altman DG.
Trials. 2015;16:57. doi: 10.1186/s13063-015-0584-6. PMID: 25886370

Medical journal editors lacked familiarity with scientific publication issues despite training and regular exposure
Victoria S.S. Wong, Michael L. Callaham
Jour Clin Epidemiology. 2012. p247–252. DOI:

Emergency medicine journal impact factor and change compared to other medical and surgical specialties
Joshua C. Reynolds MD, James J. Menegazzi PhD, Donald M. Yealy MD
Acad Emerg Med v19, i11, Nov 2012 p1248-1254 DOI:

Does mentoring new peer reviewers improve review quality? A randomized trial
Debra Houry, Steven Green, Michael Callaham
BMC Med Educ. 2012; 12: 83. Published online 2012 August 28. doi: 10.1186/1472-6920-12-83
PMCID: PMC3494517

The presentation of continuous outcomes in randomized trials: an observational study.
Schriger DL, Savage DF, Altman DG.
BMJ. 2012;345:e8486. PMID 23249670

Forest plots in reports of systematic reviews: a cross-sectional study reviewing current practice.
Schriger DL, Altman DG, Vetter JA, Heafner T, Moher D.
Int J Epid Int J Epid 2010;39:421-9.

Training of Peer Reviewers: Validation of a 5-Point Rating Scale
Michael Callaham
PLoS Med. 2007 April; 4(4): e166. Published online 2007 April 24. doi: 10.1371/journal.pmed.0040166
PMCID: PMC1876423

Training and Experience of Peer Reviewers: Authors' Reply
Michael Callaham
PLoS Med. 2007 March; 4(3): e145. Published online 2007 March 27. doi: 10.1371/journal.pmed.0040145
PMCID: PMC1831755

The Relationship of Previous Training and Experience of Journal Peer Reviewers to Subsequent Review Quality
Michael L Callaham, John Tercier
PLoS Med. 2007 January; 4(1): e40. Published online 2007 January 30. doi: 10.1371/journal.pmed.0040040
PMCID: PMC1796627

The use of the World Wide Web by medical journals in 2003 and 2005: an observational study.
Schriger DL, Ouk S, Altman DG.
Pediatrics. 2007;119:e53-60.

A normative model of peer review: qualitative assessment of manuscript reviewers’ attitudes towards peer review
Tercier, John, & Callaham, Michael L.
(2007). UC San Francisco: Department of Emergency Medicine. (106 downloads as of 8/18)

Peer Review: Consensus and Contradiction, a Qualitative Approach
Tercier, John, & Callaham, Michael L.
(2007). UC San Francisco: Department of Emergency Medicine. (74 downloads as of 8/16)

The availability of references and the sponsorship of original research cited in pharmaceutical advertisements.
Cooper RJ, Schriger DL.
CMAJ. 2005;172:487-91.

Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals
Michael Callaham, MD; Robert L. Wears, MD, MS; Ellen Weber, MD
JAMA. 2002;287(21):2847-2850. doi:10.1001/jama.287.21.2847

Author perception of peer review: impact of review quality and acceptance on satisfaction
Ellen J. Weber, MD; Patricia P. Katz, PhD; Joseph F. Waeckerle, MD; et al Michael L. Callaham, MD
JAMA. 2002;287(21):2790-2793. doi:10.1001/jama.287.21.2790

Effect of written feedback by editors on quality of reviews: two randomized trials
Michael L. Callaham, MD; Robert K. Knopp, MD; E. John Gallagher, MD
JAMA. 2002;287(21):2781-2783. doi:10.1001/jama.287.21.2781

Problems with current methods of data analysis and reporting, and suggestions for moving beyond incorrect ritual.
Schriger DL.
Eur J Emerg Med. 2002;9:203-7.

Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting
Michael L. Callaham, MD; Robert L. Wears, MD; Ellen J. Weber, MD; et al Christopher Barton, MD; Gary Young, MD
JAMA. 1998;280(3):254-257. doi:10.1001/jama.280.3.254

Unpublished research from a medical specialty meeting: why investigators fail to publish
Ellen J. Weber, MD; Michael L. Callaham, MD; Robert L. Wears, MD; et al Christopher Barton, MD; Gary Young, MD
JAMA. 1998;280(3):257-259. doi:10.1001/jama.280.3.257

Masking author identity in peer review: What factors influence masking success?
Mildred K. Cho, PhD; Amy C. Justice, MD, PhD; Margaret A. Winker, MD; et al Jesse A. Berlin, ScD; Joseph F. Waeckerle, MD; Michael L. Callaham, MD; Drummond Rennie, MD; and the PEER Investigators
JAMA. 1998;280(3):243-245. doi:10.1001/jama.280.3.243

Reliability of editors' subjective quality ratings of peer reviews of manuscripts
Michael L. Callaham, MD; William G. Baxt, MD; Joseph F. Waeckerle, MD; et al Robert L. Wears, MD
JAMA. 1998;280(3):229-231. doi:10.1001/jama.280.3.229

Time to publication of studies was not affected by whether results were positive
Michael L Callaham, Ellen Weber, Gary Young, Robert Wears, Chris Barton
BMJ. 1998 May 16; 316(7143): 1536.
PMCID: PMC1113175