Annals of Emergency Medicine
Volume 38, Issue 5 , Pages 518-526, November 2001

Evaluating the quality of systematic reviews in the emergency medicine literature☆☆☆★★★♢

Presented at the Society for Academic Emergency Medicine annual meeting, Boston, MA, May 1999.

From the Division of Emergency Medicine, Faculty of Medicine and Dentistry,* and John W. Scott Health Sciences Library,‡ University of Alberta, Edmonton, Alberta, Canada

Received 15 August 2000; received in revised form 23 January 2001 and 30 January 2001; accepted 27 February 2001.

Address for correspondence: Brian H. Rowe, MD, MSc, Division of Emergency Medicine, 1G1.63 Walter McKenzie Health Centre, 8440-112 St. Edmonton, Alberta, Canada, T6G 2B7; 780-407-6707, fax 780-407-3314; E-mail brian.rowe@ualberta.ca

Abstract 

Study Objective: The objective of this study was to examine the scientific quality of systematic reviews published in 5 leading emergency medicine journals. Methods: MEDLINE and EMBASE databases were electronically searched to identify published systematic reviews. Searches were only conducted in emergency medicine journals during the past 10 years; 4 of the journals were also hand searched. Potential reviews were assessed independently by 2 reviewers for inclusion. Data regarding methods were extracted from each review independently by 2 reviewers. All systematic reviews were retrieved and rated for quality by using the 10 questions from the overview quality assessment questionnaire. Results: Twenty-nine reviews were identified from more than 100 citations. The overall scientific quality of the systematic reviews was low (mean score, 2.7; 95% confidence interval 2.1 to 3.2; maximum possible score, 7.0). Selection and publication biases were rarely addressed in this collection of reviews. For example, the search strategies were only identified in 9 (31%) reviews, whereas independent study selection (6 [21%]) and quality assessment of included studies (9 [31%]) were infrequently performed. Overall, the majority of reviews had extensive flaws, and only 3 (10%) had minimal flaws. Conclusion: The results of the study indicate that many of the systematic reviews published in the emergency medicine literature contain major flaws; reviews with poor methodology may limit the validity of reported results. Further efforts should be made to improve the design, reporting, and publication of systematic reviews in emergency medicine. [Kelly KD, Travers A, Dorgan M, Slater L, Rowe BH. Evaluating the quality of systematic reviews in the emergency medicine literature. Ann Emerg Med. November 2001;38:518-526.]

 

☆ Dr. Kelly is currently affiliated with the Department of Rural Health, University of Melbourne, Shepparton, Victoria, Australia.

☆☆ Author contributions are provided at the end of this article.

★ This study was supported by the Division of Emergency Medicine, University of Alberta (Dr. Rowe and Dr. Travers) in Edmonton, Alberta, Canada. Dr. Kelly has received funding from the Research Excellence Envelope in the Faculty of Medicine and Dentistry at the University of Alberta.

★★ The authors declare no known conflicts of interest. Dr. Kelly, Dr. Travers, and Dr. Rowe are active members of the Airway Review Group of the Cochrane Collaboration. Dr. Rowe is an author of one of the systematic reviews included in this study. Dr. Rowe has received research funding in the past from the following industry sponsors: Astra Pharma, Glaxo Wellcome, Pharmacia and Upjohn, Trudell, and Janssen-Ortho, but is not a paid consultant to any of these companies.

♢ Reprints not available from the authors.

PII: S0196-0644(01)50890-2

doi:10.1067/mem.2001.115881

Refers to article:

  • Finding truths in clinical medicine: Through the looking glass—cracked

    David L. Schriger
    Annals of Emergency Medicine November 2001 (Vol. 38, Issue 5, Pages 566-569)

Annals of Emergency Medicine
Volume 38, Issue 5 , Pages 518-526, November 2001

Article Tools