Annals of Emergency Medicine
Volume 36, Issue 2 , Pages 149-155, August 2000

The clinician and the medical literature: When can we take a shortcut?☆

  • Peter C. Wyer, MD

      Affiliations

    • Division of Emergency Services, Columbia University College of Physicians and Surgeons, New York, NY
    • ,
  • Brian H. Rowe, MD, MSc

      Affiliations

    • Division of Emergency Medicine, University of Alberta and the Capital Health Authority, Edmonton, Alberta, Canada
    • ,
  • Gordon H. Guyatt, MD, MSc

      Affiliations

    • Department of Clinical Epidemiology and Biostatistics, McMaster University, Hamilton, Ontario, Canada
    • ,
  • William H. Cordell, MD

      Affiliations

    • Emergency Medicine and Trauma Center, Methodist Hospital, Clarian Health Partners, and Indiana University School of Medicine, Indianapolis, IN.

Received 2 February 2000; accepted 8 February 2000.

Address for correspondence: Peter C. Wyer, MD, Division of Emergency Medicine, PH 137, New York Presbyterian Hospital, 622 West 168th Street, New York, NY 10032; E-mail: pwyer@worldnet.att.net.

Article Outline

Abstract 

[Wyer PC, Rowe BH, Guyatt GH, Cordell WH. The clinician and the medical literature: when can we take a shortcut? Ann Emerg Med. August 2000;36:149-155.]

 

Back to Article Outline

Introduction 

The renowned early 20th century political satirist Will Rogers was famous for emphasizing that “All I know is what I read in the papers.” When can you believe what you read? Will Rogers was not sure, but was particularly fond of highlighting potentially outrageous implications of believing anything or everything.

The Annals of Emergency Medicine introduced its Evidence-Based Emergency Medicine (EBEM) feature as an educational series aimed at familiarizing emergency clinicians with skills and methods required to practice evidence-based medicine and to effectively use evidence-based resources.1 An EBEM installment, in addition to presenting a review of a particular topic, offers an “educational prescription” to readers to follow a similar course in conducting literature reviews and appraisals in response to questions and problems arising from their own care of emergency patients.

EBEM poses unique and perhaps unfamiliar challenges. In contrast to educational prescriptions that emergency clinicians are likely to have encountered, EBEM requires familiarity and preliminary mastery of a unique lexicon and methodology. Most physicians have not encountered this methodology in undergraduate or postgraduate training. These are contexts in which questions about whether an answer found in a textbook or clinical update review is “valid” or “believable” are generally not addressed. Medical learners are, rather, conditioned, passively, if not actively, to accept that “if it is in the book, it is true.” EBEM comes with a price and a cost. It starts out with the apparent disadvantage that it insists that clinicians attend to matters of “believability,” “validity,” and “bias.”

Evidence-based medicine is a newcomer to the stage of medical practice,2 and is controversial. The need to acquire new skills is one of its acknowledged limitations.3 We will not reiterate the pros and cons of evidence-based medicine here.3, 4, 5, 6, 7, 8 Rather, the purpose of this “perspective piece” from 3 proponents of evidence-based medicine is to help readers answer the question: “To what extent can I believe evidence from 3 types of reviews: traditional reviews, rigorous systematic views, and limited reviews of a clinical topic?” (Table).

Table. Characteristics of 3 types of reviews.
CharacteristicNarrative ReviewSystematic ReviewShortcut Review
IntentReviews current knowledge and practice in a clinical topic areaReviews all available evidence on a focused question for the purpose of deriving a single best estimate of the true resultSelects the best or most applicable studies to maximize the role of evidence from research in clinical decisionmaking
TopicClinical topic areaFocused clinical research questionQuestion arising from a practitioner’s patient care
AuthorAn established expert or researcher on the topicA team including methodologic and clinical expertiseBusy clinicians (either academic or nonacademic)
SearchingFrequently extensive, aimed at gathering literature on the topic area in questionExtensive, aimed at locating all primary studies relevant to a single research questionLimited search aimed at locating the strongest evidence addressing the question
Level of rigorCriteria not uniform because of the relatively broad scope of the reviewFollows established published criteria for research investigations of this typeFollows the methods of systematic reviews often with substantial limitations
AdvantagesProvides an overview of a defined clinical topic area for practitioners needing primary information or an updateProvides an estimate of the “true answer” based on available clinical research pertaining to a focused research questionProvides an estimate of the “true answer” but with a weaker inference than a systematic review
DisadvantagesDoes not provide the definitive answers to specific focused questionsRequires specialized expertise and a formal research protocol to prepare; results relevant to a single question onlyArticle search and selection may miss key evidence; methods of integrating discordant studies not rigorously addressed; results relevant to a single question only

In pursuing our objective, we will emphasize that Annals EBEM installments reflect a limited application of evidence-based medicine methodology.9 They are not alone in this respect. Reviews following evidence-based methodology in a limited fashion have recently become abundant in various venues, ranging from peer-reviewed publications10 to on-line “journal clubs” and “critically appraised topic (CAT) banks.” Such resources might be termed “shortcut reviews.” Reviews performed by readers, following the educational prescription of the Annals EBEM series, are examples of this kind of review.

We will begin by considering the differences between traditional reviews and rigorous systematic reviews. We will then examine the methods used in the EBEM installments and will provide readers with a “behind-the-scenes” glimpse of how these articles are produced. In closing, we will offer “users’ guide” criteria for incorporating the results of EBEM topic reviews, as well as other shortcut reviews, including practitioners’ own reviews, into clinical decisionmaking (Figure).

You might also view these criteria as self-evaluation tools for those who wish to incorporate these methods into their practices.

Back to Article Outline

How individual studies are selected and incorporated into reviews 

There are many approaches to summarizing evidence from the medical literature. Although there is wide variability in the nomenclature used for different types of reviews, “narrative review” is the term most commonly used to refer to literature reviews aimed at providing readers with a broad overview of a clinical topic.11 “Collective review” is another term that is frequently encountered in the emergency medicine literature. Although occasionally used to designate systematic reviews,12 it is more frequently used to describe a narrative review.13, 14, 15, 16, 17, 18

Narrative reviews often cover issues such as pathophysiology, epidemiology, diagnosis, prognosis, treatment, and future directions. Authors of narrative reviews attempt to synthesize both the primary evidence and prevailing practice pertaining to the topic areas they consider. Typically, there is little that is systematic about the identification, choice, critique, or summarization of the evidence. Typically also, they go beyond the presentation of evidence and provide recommendations for practice.

The term “systematic review” denotes reviews that address a focused clinical question and incorporate strategies to reduce bias. Many, but not all, systematic reviews pool quantitative data derived from individual studies. In such cases, systematic reviews may also be referred to as “meta-analyses.”

Clinicians confronted by a problem in a clinical area with which they are less familiar, or seeking an update in such an area, appreciate the availability of a recent narrative review. Clinicians with a strong foundation of knowledge and experience in an area who are searching for the answer to a specific patient care question may seek a systematic review.

What makes a systematic review systematic?19 First, the investigators develop explicit eligibility criteria for their studies, including the patient population, a specific intervention or exposure and a comparison, the outcome measures, and the range of acceptable methodology. Second, the investigators then conduct a systematic and comprehensive search for all studies that meet their eligibility criteria. In the most rigorous systematic reviews, the investigators search thoroughly for unpublished and foreign language literature.

Third, having identified potentially eligible studies, the investigators apply their eligibility criteria to determine which studies to include. Having identified eligible studies, they systematically assess the methodologic quality of the studies they have chosen. Two aspects of this process require particular judgment: the application of eligibility criteria and the assessment of methodologic quality. Ideally, investigators will conduct these evaluations in duplicate or triplicate and will report the extent of agreement.

Fourth, the reviewers then abstract the data and, if the studies are sufficiently similar in design, population, intervention, and outcome assessment, they will pool the data across studies. They will look carefully for important differences in outcome across studies, and reasons for those differences. In the end, they will provide an unbiased, precise estimate of the effect of the intervention or exposure on the outcome of interest in the population studied. Sophisticated reviewers, understanding that moving from evidence to action requires a value judgment, will provide only evidence and refrain from offering recommendations.20

The authors of narrative reviews do not generally describe the details of the literature searches or of the other methods they used in preparing their articles. Reflecting the influence of evidence-based medicine, and its concerns regarding review methodology, some narrative reviews are beginning to include such information.21 However, the 2 types of review remain readily distinguishable by virtue of their different goals. Systematic reviews are becoming more common in the medical literature and can be specifically accessed through resources such as the Cochrane Library.22 Methodologists have offered standards for systematic reviews,11, 19, 23 and a recent installment in the EBEM series has illustrated the application of these standards.24

Systematic reviews address a single, focused, clinically relevant research question and may restrict themselves to studies reflecting a strong methodology. They may therefore admit a small number of studies, or even a single relevant study, into their reviews. Whether clinicians are confident in basing practice on a systematic review in such a situation will depend on a multiplicity of factors. These include the methodologic quality of the study or studies, the precision of the results, whether the studies involved populations sufficiently similar to their own patients, and whether the studies examined the impact of alternative management strategies on all relevant outcomes.

Back to Article Outline

Systematic reviews as the methodologic standard for shortcut reviews 

We are now ready to consider the question: “How do the methodologic standards of systematic reviews inform the Annals EBEM reviews, and other shortcut reviews?” We are taking Annals EBEM reviews as a “case in point” for this discussion, precisely because we, as editors and advisors to this effort, are familiar with the concepts and processes behind this series. We will begin by summarizing the goals and premises of the Annals EBEM feature and by offering some examples drawn from the published installments in that series.

Annals EBEM reviews are designed to address what a busy clinician might do to efficiently incorporate evidence from clinical research into decisionmaking for individual patients. Evidence-based clinicians are best advised to begin by determining whether someone has already done the hard work, and done it well. In other words, if there is any possibility that one might be available, the clinician should begin by seeking a systematic review. If a systematic review is not available, clinicians must quickly identify the best primary studies, and use these to inform their decisions.

How thoroughly do Annals EBEM reviews follow the prescription for a systematic view? The answer is: “As much as possible, considering the limitations of the form.” EBEM authors adduce explicit eligibility criteria, including specifying the patient population, the intervention or exposure, the outcomes of interest, and the optimal study methodology, all of which arise from consideration of a specific patient. Their searches are less comprehensive than in a systematic review, and their eligibility criteria may be applied by only a single individual. The limited search and the possibly idiosyncratic application of the eligibility criteria in an EBEM review increases, relative to a systematic review, the likelihood of a biased selection of studies. The following 2 examples illustrate approaches taken by authors of published Annals EBEM reviews in selecting articles for inclusion.

In an EBEM review on thrombolytic agents in stroke,25 the review was restricted to randomized trials using intravenous recombinant tissue plasminogen activator (rt-PA), despite the existence of several trials using streptokinase. The authors believed that the negative results of the streptokinase trials were well known in the emergency medicine community and that clinicians would therefore consider the issue of the efficacy of rt-PA to be the only relevant question. The EBEM authors’ selection criteria also excluded studies that required, as part of their protocols, carotid angiography before administration of rt-PA. Immediate carotid angiography is a procedure that is not available to stroke patients presenting to emergency departments in most community hospitals. Only 2 randomized controlled trials satisfied the authors’ inclusion criteria. The authors also examined a Cochrane review26, 27 and one other systematic review28 to help ensure that they had not missed any eligible studies in their MEDLINE search. However, because they had included studies involving prior angiography in the pooled results, the systematic reviews were not chosen as the primary studies to be considered in the EBEM review. The authors of the EBEM review, following the model of what a clinician would have to do in the same circumstances, therefore reviewed individually the 2 studies that conformed to the question they had originally posed.

In the installment on the usefulness of the peripheral WBC count in predicting the likelihood of acute appendicitis,29 the review was restricted to studies reporting data on multiple WBC count cutoff values for “positivity” and “negativity.” The authors made this decision on the basis of a consideration of the potential pitfalls of using single cutoffs for continuous variables.30, 31 Following the premises of the EBEM feature, they elected not to attempt a systematic review that might take into account the results of the many studies reporting the performance of the WBC count in this condition. Only 2 of the identified studies found in the authors’ search fulfilled the inclusion criteria, and only 1 of them provided sufficient data on the multiple WBC count cutoffs.

What about unpublished studies? Is it possible that a study that was never submitted for publication, that was rejected, or that is still under way might observe a completely different result from that reported in an EBEM review? In general, studies reporting a negative result of a therapeutic intervention are less likely to be published than are those reporting a positive result.32 Pressure from the pharmaceutical industry may contribute to this inequity.33 This “publication bias” is particularly likely to play a role when only small studies have been performed. Large well-funded studies are more likely to be published irrespective of their results. An EBEM review on topical ophthalmic nonsteroidal anti-inflammatory drugs for corneal abrasion might have been susceptible to publication bias.34 The authors did not use hand-searching, nor did they poll researchers or pharmaceutical industry sponsors of relevant research to identify unpublished studies. They did, however, report their results to be equivocal on the basis of the studies they included, and they also warned of their methodologic limitations.

In the case of Annals EBEM reviews, which involve a more limited search strategy than used for systematic overviews, the problem is not only that of unpublished studies, but also of published studies that might have been missed in the search. There are many ways in which “missed” studies might threaten the conclusions of a shortcut review. If only a few small studies were included, and if a number of potentially eligible, small, unpublished, or foreign language studies were not detected by the search strategies used, the results of the review might differ from those that would have been obtained had those studies been included.

Statistical pooling of the results, an approach used in many systematic reviews of therapeutic or diagnostic interventions, is beyond the skill of all but a few practicing clinicians. Similarly, an exhaustive search, on the level of what is required for a valid meta-analysis, is beyond the resources of most clinicians. Because of this, EBEM authors are encouraged to establish a high-quality standard for admitting studies into Annals EBEM reviews. One diagnosis study was excluded from consideration because the study population was confined largely to patients with positive findings at surgery.29 One therapy study was considered but dismissed because only 56% of patients were followed to the point that the study outcomes could be determined.34

The process of planning an EBEM installment encompasses several measures aimed at compensating for the methodologic limitations. EBEM review topics are chosen with foreknowledge that evidence exists relevant to the question that has been posed, and the decision to pursue a particular topic is based on a preliminary knowledge of the nature of the studies in question. The clinical scenario is carefully framed to reflect the reality of emergency medicine practice and in many cases is based on real patient encounters.

When the review concerns an issue of therapy, or when a controlled trial is likely to provide the “best evidence” pertaining to the question that has been asked, the reviewer consults special databases, such as the Cochrane Library.35 The reviewer routinely seeks published systematic views and treats them as the highest level of evidence. The search protocols correspond to the searches actually performed by the authors, and they are described in a fashion that allows them to be reproduced and updated. Although more rigorous than is usual in current clinical practice, the shortcut reviews do reflect an approach that frontline practitioners can attain.

In the course of peer review, the manuscripts of EBEM installments are evaluated by a member of the Annals panel of consultants possessing special expertise in the clinical area in question. This review lessens the likelihood that important published studies have been overlooked. Expert review may point out ongoing studies that will alert readers to the need for updating the EBEM review in the future. The peer review protocol for EBEM installments also includes consultation with at least 2 librarians chosen from a panel made up of individuals trained in evidence-based searching methods. This acts as a further control over the likelihood that important includable studies were missed in the authors’ search.

Although these controls are part of the Annals EBEM production protocol, they do not in themselves guarantee the validity of any particular review. Readers must ultimately decide for themselves whether any particular EBEM installment, or any other shortcut review, adequately meets minimum quality standards of an evidence-based review. In this regard, readers, in the course of reading an EBEM review, might ask themselves, “Is this the question I would have asked in relationship to the clinical scenario as described?” This question highlights an important limitation of evidence-based reviews: they are completely specific to the question that has been asked. The more specific the question, the less the conclusion can be generalized to a different, even closely related, question.

In summary, it is important that all clinicians understand, and we hope that the Annals experience helps illustrate, the ways that bias may be introduced into a limited shortcut review, as well as the measures that may be taken to minimize the effects of such bias.

Back to Article Outline

How should clinicians decide when a review is “good enough to use”? 

A commitment to evidence-based medicine demands that clinicians use the strongest evidence they can find to guide their clinical decisions. If a systematic review is not available, clinicians must rely on their own less than systematic reviews of primary studies located and selected in the course of their own searches, understanding at the same time the potential liabilities of publication bias and of failing to find relevant published studies. How can clinicians determine what confidence they should place in shortcut reviews in general, and Annals shortcut reviews in particular?

We have already outlined the special resources available to Annals EBEM authors and editors. Not only do they enjoy the input from librarians and experts, but you can be sure they spend much more time in preparing their review than you would have available as part of routine clinical care. One would expect that their results would therefore be less open to the biases of an unsystematic search and selection of studies than would frontline clinicians’ shortcut reviews.

We offer a qualification regarding reliance on experts. Experts often develop strong opinions based on their personal experience, their own research, or the sources of their funding. Indeed, one of the arguments in favor of systematic reviews is the frequency with which, in the past, expert recommendations have been inconsistent with the evidence.36 One study even found an inverse correlation between self-rated expertise and the quality of a review.37 Hopefully, as experts gain a better understanding of, and deeper respect for, evidence-based methods, the extent of these biased assessments will decrease. Indeed, textbooks that both rely on experts and strive toward evidence-based methods are already providing useful shortcut reviews. In the meantime, if you are on your own, be careful if you consider yourself an “expert”!

The Figure presents criteria that might be used to judge the likelihood that limitations of the searching and study selection procedures of a shortcut review yielded biased estimates. If, as in the case of a recent EBEM installment on ipratropium bromide for pediatric asthma,24 the EBEM review presents the results of a rigorously conducted systematic review, you are on very strong ground. If the review offers the results of one or more extremely large studies (thousands of patients), inferences are also strong. The reason is that a number of small studies that the reviewers are likely to have missed are unlikely to substantially modify the best estimate of the treatment effect (and the reviewers are unlikely to have missed other large studies). A large number of small studies also provide moderately strong inferences: unless the reviewers have missed an equally large number of studies, the results may withstand additional undetected evidence. When only a few small studies are available, the likelihood of bias is high.

We close with a reminder that EBEM reviews are intended not only to provide summaries of evidence on important emergency medicine clinical issues, but, perhaps even more importantly, to demonstrate application of a model for addressing such. If our shortcut summaries also happen to yield strong inferences about the questions they address, so much the better.

Back to Article Outline

References 

  1. Waeckerle JF, Cordell WH, Wyer P, et al.  Evidence-based emergency medicine: integrating research into practice. Ann Emerg Med. 1997;30:626–628
  2. Guyatt GH. Evidence-based medicine [editorial] ACP J Club. Ann Intern Med. 1991;114(Suppl 2):A-16; 1991
  3. Rosenberg W, Donald A. Evidence based medicine: an approach to clinical problem-solving. BMJ. 1995;310:1122–1126
  4. Feinstein AR, Horwitz RI. Problems in the “evidence” of “Evidence-based Medicine.”. Am J Med. 1997;103:529–535
  5. Evidence-based medicine, in its place. Lancet. 1995;346:785; [editorial]
  6. Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet. 1995;345:840–842
  7. Molesworth N. Down with EBM!. BMJ. 1998;317:1720–1721
  8. Grahame-Smith D. Evidence based medicine: Socratic dissent. BMJ. 1995;310:1126–1127
  9. Guyatt GH. Evidence-based emergency medicine. [editorial] Ann Emerg Med. 1997;30:675–676
  10. Carley SD, Mackway-Jones K, Jones A, et al.  Moving towards evidence based emergency medicine: use of a structured critical appraisal journal club. J Accid Emerg Med. 1998;15:220–226
  11. Oxman AD, Cook DJ, Guyatt GH, et al.  Users’ guides to the medical literature: VI. How to use an overview. JAMA. 1994;272:1367–1371
  12. Young KD, Seidel JS. Pediatric cardiopulmonary resuscitation: a collective review. Ann Emerg Med. 1999;33:195–205
  13. Freedland ES, McMicken DB. Alcohol-related seizures, part II: clinical presentation and management. J Emerg Med. 1993;11:605–618
  14. Freedland ES, McMicken DB. Alcohol-related seizures, part I: pathophysiology, differential diagnosis and evaluation. J Emerg Med. 1993;11:463–473
  15. Go GW, Baraff LJ, Schriger DL. Management guidelines for health care workers exposed to blood and body fluids. Ann Emerg Med. 1991;20:1341–1350
  16. Dickey LS. Diving injuries. J Emerg Med. 1984;1:249–262
  17. Davis SM. Antishock trousers: a collective review. J Emerg Med. 1986;4:145–155
  18. Rabold M. High-altitude pulmonary, edema: a collective review. J Emerg Med. 1989;7:426–433
  19. Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Assoc J. 1988;138:697–703
  20. Guyatt GH, Sinclair J, Cook DJ, et al.  Users’ guides to the medical literature: XVI. How to use a treatment recommendation. JAMA. 1999;281:1836–1843
  21. Fontanarosa PB. An evidence based approach to diagnostic testing in emergency medicine. Emerg Med Clin North Am. 1999;17:1–8
  22. Cochrane Library (issue 4). Oxford: , United Kingdom: Update Software; 1999;
  23. Mulrow CD, Oxman AD. The Cochrane Collaboration Handbook (version 3.0.2). The Cochrane Library. Oxford, United Kingdom: : Update Software; 1997;
  24. Rowe BH, Travers AH, Holroyd BR, et al.  Nebulized ipratropium bromide in acute pediatric asthma: does it reduce hospital admissions among children presenting to the emergency department?. Ann Emerg Med. 1999;34:75–85
  25. Wyer PC, Osbom HH. Recombinant tissue plasminogen activator: in my community hospital ED, will early administration of rt-PA to patients with the initial diagnosis of acute ischemic stroke reduce mortality and disability?. Ann Emerg Med. 1997;30:629–638
  26. Wardlaw JIM, Warlow CP, Counsell C. Systematic review of evidence on thrombolytic therapy for acute ischaemic stroke. Lancet. 1997;350:607–614
  27. Wardlaw JM, Yamaguchi T, del Zoppo GJ. The efficacy and safety of thrombolytic therapy in acute ischaemic stroke: a systematic review of the randomized trials comparing thrombolysis with control. Cochrane Database of Systematic Reviews. Oxford, United Kingdom: : Update Software; 1997;
  28. Wardlaw JM, Warlow CP. Thrombolysis in acute ischemic stroke: does it work?. Stroke. 1992;23:1826–1839
  29. Snyder BK, Hayden SR. Accuracy of leukocyte count in the diagnosis of acute appendicitis. Ann Emerg Med. 1999;33:565–574
  30. Tandberg D, Deely JJ, O’Malley AJ. Generalized likelihood ratios for quantitative diagnostic test scores. Am J Emerg Med. 1997;15:694–699
  31. Carey M, Brennan J, Ciconelli R, et al.  Likelihood ratios when the results of diagnostic tests are continuous variables. [letter] Am J Emerg Med. 1998;16:722–723
  32. Callaham ML, Wears RL, Weber EJ, et al.  Positive-outcome bias and other limitations in the outcome of research abstracts submitted to a scientific meeting. JAMA. 1998;280:254–257
  33. Rennie D. Thyroid storm. [editorial] JAMA. 1997;277:1238–1243
  34. Brown MD, Cordell WH, Gee AS. Do ophthalmic nonsteroidal anti-inflammatory drugs reduce the pain associated with simple corneal abrasion without delaying healing?. Ann Emerg Med. 1999;34:526–534
  35. Rowe B, Alderson P. The Cochrane Library: A resource for clinical problem solving in emergency medicine. Ann Emerg Med. 1999;34:86–90
  36. Antman EM, Lau J, Kupelnick G, et al.  A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments of myocardial infarction. JAMA. 1992;268:240–248
  37. Oxman AD, Guyatt GH. The science of reviewing research. Ann N Y Acad Sci. 1993;703:125–133

☆ Reprints not available from the authors.

PII: S0196-0644(00)06509-4

doi:10.1067/mem.2000.108656

Annals of Emergency Medicine
Volume 36, Issue 2 , Pages 149-155, August 2000

Article Tools

* Image Usage & Resolution