Current Status of Peer Review at Annals of Emergency Medicine

Introduction 

As a peer-reviewed scientific journal, Annals of Emergency Medicine is reliant on its volunteer team of expert reviewers to assess the suitability of articles for publication. In 2005, Annals received 1,231 total submissions, of which 521 underwent external peer review, including 67% of the original research articles submitted. Our referees performed 1,317 total reviews, an average of 2.5 per reviewed article.

Although the peer-review process lies at the heart of the dissemination of science, many authors and readers find it mysterious. In this article, we wish to summarize the current system of peer review at Annals, answer the most common questions we receive, and describe recent progress in our review process since our adoption of an electronic article processing system (Editorial Manager; Aries Systems Corporation, North Andover, MA) in January 2002.

Who are our peer reviewers? 

There are currently 1,034 individuals who serve as Annals reviewers. Although almost all are emergency physicians, a minority are carefully selected specialists from other fields with overlapping expertise. We also have 4 research methodology (ie, “methodology and statistics”) editors, one of whom evaluates every original research article before publication. We also have 5 librarian reviewers with special proficiency in literature search strategies to aid in our evaluation of systematic reviews and meta-analyses.

How does one become a reviewer for Annals? 

Potential new reviewers for Annals may be nominated by an editor or may contact the journal, directly expressing this interest. In either of these situations, candidates must demonstrate through their curriculum vitae a minimum of 2 first-author original research articles relating to emergency medicine published in either Annals or in journals of similar or higher impact or demonstrate evidence of equivalent personal involvement in research. Our approach is to encourage new reviewers to apply but to require them to have successfully performed the task (of authorship) they are being asked to judge. Exceptions have been made for obvious experts in focused areas (eg, bioethics, emergency medicine administration) in which original research is not consistently the primary format for advancement of knowledge.

New reviewers can also be informally recruited if an Annals editor decides that a specific individual might be useful for a given article according to their knowledge of the reviewer or their research. Such invitations have no further stipulations other than the judgment of the editor in question.

New reviewers are directed to our Internet-based training module, “An Instructional Guide for Peer Reviewers of Biomedical Manuscripts.”1 This module is available at no charge to nonreviewers as well and provides detailed instructions about the review process (in the form of short audio/slide lectures on pertinent topics) and our expectations for quality reviews, with examples.

Typically, new appointments to the editorial board at Annals are selected from the pool of our very best peer reviewers.

How are reviewers selected for specific articles? 

When new articles are received electronically through our Editorial Manager software, they are first inspected by a staff member in the Annals office to ensure that all necessary elements (eg, cover letter, author contributions statement, CONSORT checklist if applicable) are in place. On verification of the article’s adherence with journal requirements, this staff member then electronically routes the article to 1 or more Annals editors with special expertise in the given content area. We organize emergency content into a classification tree, with 21 major categories and 99 subcategories, and editors and reviewers are linked to various items according to their expertise and identified using this structure.

Some articles judged lowest priority for publication will be rejected quickly by the editor or editors involved (because of lack of interest to the Annals audience or obviously inadequate design or article quality); however, most of the original research submissions will be sent for formal peer review. Reviewers are typically selected by the supervising editor, who uses 3 methods. First, our editors are usually quite familiar with the cadre of reviewers specialized in their content areas and use this personal knowledge to match their expertise to specific articles. Second, potential reviewers are often located through review of the article’s reference list or by an editor literature search on the article topic. The final common method for reviewer identification is using our article processing software to suggest reviewers. To accomplish this, the software counts the number of content classification matches between the individual reviewers and the article itself and lists those individuals with the most overlap. A consequence of this latter mechanism is that the more content classifications selected by any given reviewer, the more likely that he or she will be matched to articles and thus invited to review.

Are reviews blinded and, if so, why? 

There is no evidence that blinding articles improves the quality of the peer-review performed, thus supporting the longstanding tradition that most general medicine and many specialty journals do not blind their peer review.2, 3

However, the studies of blinding assessed only the objective quality of the review, and no study has examined whether that review was fair or accurate. Because of this important lacuna in our knowledge, Annals blinds both authors and reviewers. We believe that it helps decrease the potential for bias and increases the likelihood of fairness for the authors. Because Annals also blinds the author to the reviewer’s identity (another longstanding tradition at most journals), it seems only fair that the reverse be true as well.

Why does Annals have specific methodology and statistics Editors? 

Before any original research article can be accepted for publication, it must be analyzed by one of our 4 methodology and statistics editors: Richelle J. Cooper, MD, MSHS; Margaret Hsieh, MD; David L. Schriger, MD, MPH; and Robert L. Wears, MD, MS. These individuals have graduate-level training in statistics or epidemiology and well-recognized statistical expertise. In 2005, our methodology and statistics editors together completed a remarkable 338 total article reviews.

Annals is unusual among journals in requiring such specific scrutiny of the methodology and statistics in submitted articles, and this means that our authors are more likely to be questioned about their study design and analysis than they might if they submitted their articles elsewhere. However, we believe that this process provides a vital safeguard to ensure the consistent rigor of our published research. We have found (from formal study),4, 5, 6 as have many other major journals, that reviewers without special statistical interest and training do not detect many important design and methodology weaknesses, and therefore the suggestions of our methodology editors lead to important revisions that improve the quality of accepted articles.

Our methodology reviewers actively collaborate to devise a consistent Annals philosophy on research design and statistics, which is articulated in our Instructions for Authors. Use of this small number of reviewers permits more homogenous reviews and, consequently, we believe increases fairness and even-handedness.

How does Annals assess and monitor its reviewers? 

As can only be expected with a pool of 1,034 busy volunteers, our reviewer group has always been heterogenous in the quality of their reviews and the speed with which they are submitted. Over time, many individuals substantially improve their review proficiency, whereas for others, alternative commitments prevent them from devoting as much energy to Annals.

For just over 2 decades, all completed reviews have been graded by our editors with a 5-point numerical scale (Figure 1), and average-review-quality statistics have been made available to editors selecting reviewers. The qualities of a good review have been formally defined, as have the criteria for each of the 5 ratings. Previous research at Annals has shown that this rating system is reliable in identifying higher-quality reviewers, independent of the individual article or the editor who does the rating.7 Before our adoption of an electronic article processing system, it was difficult or impossible to stratify our reviewer pool by promptness or reliability, which led to periodic situations in which the typical quantity of initial review invitations resulted in too few reviews or too cursory a review for adequate article evaluation, and substantial delays resulted from the belated procurement of additional reviews.

View Large Image Download to PowerPoint

Figure 1. Annals’ criteria for rating review quality.

With the adoption of our article processing software in January 2002, it became feasible to track statistics on individual reviewer promptness and reliability, as well as quality. Accordingly, we switched from what was essentially a single rank of reviewers to a dynamic 3-level hierarchic arrangement, ie, a highest level, a middle level, and a lower level. The objective of this stratification was to identify a subset of “best” reviewers with a maximal likelihood of promptly returning quality reviews. Maintenance of these reviewer tiers has been accomplished by running detailed reports every 6 months of reviewer data and reassigning individuals up or down the hierarchy according to predefined performance criteria. These include measures of quality (average review rating), productivity (number of reviews completed), and reliability (number of review invitations ignored or declined, number of reviews submitted late).

We believe that this tiered system, while first serving the needs of the journal, remains fair to our valued reviewers in that any given individual has the ready ability to rise to the top tier through demonstrated performance. Conversely, individuals with limited time available for the journal can remain a member of our reviewer group and be invited to review a smaller number of select articles that may better fit their specific area of expertise. Given that this stratification is a confidential internal tracking mechanism, its classification data are available only to journal editors and staff. They are not shared with reviewers because formal study at Annals has shown that, paradoxically, knowledge of one’s rating and simple feedback about improvement do not in fact provide reviewers a tool to improve their performance and do not lead to improvement in ratings (a problem common to many other popular but unproven teaching interventions in medicine).8

The result of this new system is that editors can direct the bulk of their review invitations to the subset of better-performing reviewers, and indeed the median review score assigned in 2005 was a 4. Total review times can be minimized by using this more consistently reliable group.

How well do Annals reviewers perform? 

Throughout the past few years, we have noted dramatic improvements in overall reviewer reliability. Reviews are more likely to be turned in on time (Figure 2), they are turned in faster (Table 1, Figure 3), and the probability of any given review invitation resulting in a completed review is higher (Figure 4). We are delighted with these improvements because they permit us to substantially reduce decision times for submitted articles. Readers of our annual journal performance report9 will note impressive and continued gains in timeliness during the past 4 years. Increased reviewer dedication to the journal no doubt has played an important role in these improvements, as has the ease and efficiency of using an electronic review system and our increasing ability to target most reviews to our better-performing reviewers.

View Large Image Download to PowerPoint

Figure 2. Fraction of completed reviews that were turned in on time (≤14 days).

Table 1. Number of days required for completed reviews.

	Year	Median, days	75th Percentile, days
	2002	12	16
	2003	12	14
	2004	10	14
	2005	10	13

View Large Image Download to PowerPoint

Figure 3. Distribution of days required for completed reviews in 2005.

View Large Image Download to PowerPoint

Figure 4. Fraction of total reviewer invitations resulting in a completed review.

Our reviewers are all volunteers, and we highly value the time they spend on this task for the benefit of our specialty. Performing a good peer review is an arduous and a time-consuming task, and it is also a key academic process as important to the dissemination of good science as writing an article itself. For that reason, we believe it should always be referenced and reviewed by promotion and tenure committees.

Our editors have noted improvements in review quality that parallel those noted for reliability; however, our editors grade these reviews on the curve to keep “grade inflation” from defeating the functionality of the rating system.

We try hard to publicly recognize the contributions of our reviewers. In 2005, Annals began awarding Continuing Medical Education credit for reviews performed. All active reviewers are listed annually in the journal. Additionally, we also publish annually a list of our top reviewers, ie, individuals who have distinguished themselves according to the unusual quality, volume, and timeliness of their reviews in the previous year. We are also indebted to a subset of our reviewers who exhibit extraordinary dedication to the journal and perform an unusually high number of reviews (Table 2). Senior reviewers, honored with a listing in each issue of Annals, are referees who have appeared on the annual top reviewers list twice or more the last 4 years. They are the best of the very best, and their work represents a substantial academic contribution to the journal and our specialty. As stated earlier, it is from this pool of tried and true top reviewers and senior reviewers that almost all new appointments to our editorial board originate.

Table 2. Most on-time reviews in 2005 (4 or more).

	Name	n	Name	n
	Milling, Truman John	33	Schull, Michael J.	6
	Coon, Troy P.	28	Suner, Selim	6
	Goodacre, Steve	21	Wyer, Peter C.	6
	Burton, John H.	12	Yealy, Donald M.	6
	Cone, David C.	11	Ardagh, Michael W.	5
	Mower, III, William R.	11	Barton, Chris W.	5
	Barlas, David	10	Birnbaum, Adrienne J.	5
	Bernstein, Steven L.	10	Chudnofsky, Carl R.	5
	Chan, Theodore C.	10	Dunford, James V.	5
	Guss, David A.	10	Bonsu, Bema K.	5
	Jackimczyk, Kenneth C.	10	Finnell, John T.	5
	Kaji, Amy H.	10	Gennis, Paul R.	5
	Wang, Henry E.	10	Glazer, James L.	5
	Werman, Howard A.	10	Jang, Timothy B.	5
	Bender, Paul	9	Kelly, Anne-Maree	5
	Brown, Michael D.	9	Kosowsky, Joshua M.	5
	Isakov, Alexander P.	9	Levitan, Richard M.	5
	Rinnert, Kathy J.	9	Lewin, Matthew R.	5
	Sarko, John	9	Ma, John O.	5
	Atzema, Clare	8	Marco, Catherine A.	5
	Campbell, Jack P.	8	McKay, Mary P.	5
	Gill, Michelle R.	8	O’Malley, Gerald F.	5
	Holmes, James F.	8	Overton, David T.	5
	McGeorge, Francis T.	8	Quinn, James V.	5
	Rowe, Brian H.	8	Ramoska, Edward A.	5
	Sun, Benjamin C.	8	Schmidt, Terri A.	5
	Weber, Ellen J.	8	Barry, James D.	4
	Welch, Robert D.	8	Bartfield, Joel M.	4
	Augustine, James J.	7	Brice, Jane H.	4
	De Maio, Valerie J.	7	Callaway, Clifton	4
	Dronen, Steven C.	7	Camargo, Carlos A.	4
	Jones, Alan E.	7	Counselman, Francis L.	4
	Krauss, Baruch	7	Dalsey, William C.	4
	Mothershead, Jerry L.	7	Handel, Daniel A.	4
	Silbergleit, Robert	7	Ishimine, Paul T.	4
	Sklar, David P.	7	Madsen, Troy E.	4
	Spaite, Daniel W.	7	Marill, Keith A.	4
	Woodward, George A. (Tony)	7	McHugh, Damian F.	4
	Adams, Bruce D.	6	Miner, James R.	4
	Borg, Keith	6	Newman, David H.	4
	De Lorenzo, Robert A.	6	Schafermeyer, Robert W.	4
	Gorelick, Marc H.	6	Singer, Adam J.	4
	Green, Steven M.	6	Slovis, Corey M.	4
	Greenes, David S.	6	Sztajnkrycer, Matthew D.	4
	Hendey, Gregory W.	6	Vilke, Gary M.	4
	Hsieh, Margaret	6	Williams, Robert M.	4
	Li, James	6	Witting, Michael D.	4
	Meador, Steven A.	6	Yip, Luke	4

Conclusion 

We are extremely pleased with, and grateful to, our knowledgeable and hard-working reviewers. Their exceptional diversity of expertise permits us to authoritatively evaluate articles covering the full spectrum of emergency medicine, and their overall quality and reliability are superb. They provide the expertise that helps determine the quality of what we publish and thereby play a key role in the dissemination of scientific information. We remain grateful to this talented team, without whom Annals could not function.