Annals of Emergency Medicine
Volume 36, Issue 5 , Pages 517-519, November 2000

One is the loneliest number: Be skeptical of evidence summaries based on limited literature reviews☆☆☆

University of California– Los Angeles Emergency Medicine Center, University of California– Los Angeles School of Medicine, Los Angeles, CA.

Article Outline

Abstract 

[Schriger DL. One is the loneliest number: be skeptical of evidence summaries based on limited literature reviews. Ann Emerg Med. November 2000;36:517-519.]

 

See related article, p. 520.

One is the loneliest number. —Harry Nilsson

The examples are numerous and varied. Cubism; the Heisenberg uncertainty principle; postmodernism in philosophy, literary criticism, and journalism; the OJ trial—the premise that there is a singular version of the truth is dead.1, 2 Why then, have some physicians championed evidence-based medicine (EBM), a seemingly reductionist, deductive model of medical practice, at the time when other disciplines are abandoning this mode of reasoning? Isn’t EBM fraught with the potential for oversimplification? Isn’t EBM just an overreaction to the findings of variations research and a longing by patients and practitioners for logical singular answers to complex questions?3, 4 Yet, who can be against it? EBM rings of apple pie and sliced bread. The alternative, “non–evidence-based medicine,” is unthinkable; it squawks of quackery. Surely, our practice should be informed by evidence.

Medical decisions are complex. Literature-based evidence, hunches, and feelings derived from experience, patient preference, and available resources must all be considered. Each patient and physician, when confronted by the need to make a medical decision, must determine how much each of these inputs will influence the decision. This determination requires judgment. It cannot be evidence-based. Thus, even if EBM methods consistently deliver the most unbiased estimates of the truth, medical decisions unavoidably involve matters of the heart as well as those of the head. EBM is not a panacea for the uncertainty inherent in the practice of medicine.5 Evidence-based medicine is a misnomer; evidence-informed medicine is a more accurate designation. At its best, the EBM process is invoked to inform decisions, not make them.

In this counterpoint, I focus on how basing an EBM recommendation on a single published article might compromise the quality of information offered to a medical decision. I stress that a fundamental problem with the single article approach is that it gives one piece of evidence a weight of one, and all other evidence a weight of zero. Readers will need to decide whether this all-or-nothing weighting scheme is a legitimate model for synthesizing knowledge.

Imagine a legal system that, after permitting all traditional forms of evidence to be introduced, has the judge instruct the jury to completely ignore everything but the testimony of one witness. Would we be comfortable with that process? Do we really believe that from all the primary published evidence available to us—randomized trials, cohort studies, case-control studies, case series—one piece of evidence merits a weight of 100% and all else deserves a weight of zero? Are we not better off considering all of the evidence before passing judgment?

Proponents will argue that limited, best evidence reviews are required for three reasons. First, simplified reviews are better for teaching purposes. Second, these truncated reviews better represent the kinds of EBM analyses that a physician might independently attempt. Third, it is not realistic to expect individual investigators to be comprehensive when federal agencies such as the Agency for Healthcare Research and Quality (AHRQ; formerly Agency for Health Care Policy and Research [AHCPR]) spend hundreds of thousands of dollars on each telephone book–sized evidence report.6 They justify the use of limited evidence by arguing that studies enacted with designs that are less vulnerable to bias deserve greater consideration than less robust efforts.7, 8 However, there is evidence that observational studies can be as valid as randomized control trials, and evidence that quality scoring does not reliably identify articles that better estimate the truth.9, 10, 11 There is no empiric evidence supporting the one or zero, all-or-nothing, weighting scheme used in many evidence-based emergency medicine analyses.

Simplification comes at a price. How do we know if important evidence was excluded by overly restrictive criteria? Is it not possible that the “best” study was done in an unusual setting, or with an atypical patient population, or by a less than careful investigator? As I am writing this, the principal investigator of the sole positive trial of autologous bone marrow transplant in stage III breast cancer confessed to falsifying the data.12 Proponents of this procedure had vehemently argued that the South African trial was the single most important piece of evidence. They now understand the risk of divining the truth from a single publication.

Disregarding the possibilities for fraud, what if 10 investigators independently reach the same conclusion, a conclusion that differs from the one study selected as “best” evidence? Do we really want to completely ignore the 10 studies because they did not use the “best” design? Without doing the comprehensive analysis of all reasonable evidence, there is no way of knowing whether the answer derived from a limited EBM analysis of a highly selective sample of studies will match that gleaned from the full review. When the EBM process is transformed from a comprehensive exercise to a task of selecting one or a few best studies, the likelihood that valid conclusions will be reached is diminished.

Lest cynicism run unchecked, remember that the spirit of EBM is fine. We need evidence-informed practice. We must let the evidence carve out a path of reasonable action, an interstate on which each patient-physician pair can select an appropriate lane. The number of lanes will shrink as the amount, quality, and homogeneity of the evidence increases. We need to teach physicians to use information developed through the EBM process, but such teaching should be accompanied by full disclosure of the problems and limitations of EBM and should constitute education, not indoctrination.

Evidence-based emergency medicine reviews, when based on a limited review of the evidence, must not be viewed as irrefutable representations of the truth. They are good, helpful articles. They instruct and they inform. Nevertheless, for the most part, they summarize the results of one or a handful of articles and are better at seeding debate than delineating truth. To base one’s practice solely on an evidence-based emergency medicine review is as dangerous as practicing solely on the basis of one’s experience.

Back to Article Outline

References 

  1. Hodgkin P. Medicine, postmodernism, and the end of certainty. BMJ. 1996;313:1568–1569
  2. Raithatha N. Medicine, postmodernism, and the end of certainty. Postmodern philosophy offers a more appropriate system for medicine. [letter] BMJ. 1997;314:1044
  3. Wennberg J, Gittlesohn A. Small area variations in health care delivery. Science. 1973;142:1102–1108
  4. Chassin MR, Brook RH, Park RE, et al.  Variations in the use of medical and surgical services by the Medicare population. N Engl J Med. 1986;314:285–290
  5. Fisher ES, Welch HG. Avoiding the unintended consequences of growth in medical care: how might more be worse?. JAMA. 1999;281:446–453
  6. Evidence-based practice centers: synthesizing scientific evidence to improve quality and effectiveness in clinical care. Agency for Healthcare Research and Quality Web site Available at http://www.ahrq.gov/clinic/epc/February 4; 2000; Accessed
  7. Miller JN, Colditz GA, Mosteller F. How study design affects outcomes in comparisons of therapy. II: surgical. Stat Med. 1989;8:455–466
  8. Colditz GA, Miller JM, Mosteller F. How study design affects outcomes in comparisons of therapy. I: medical. Stat Med. 1989;8:441–454
  9. Benson K, Hartz AJ. A comparison of observational studies and randomized, controlled trials. N Engl J Med. 2000;342:1878–1886
  10. Concato J, Shah N, Horwitz RI. Randomized, controlled trials, observational studies, and the hierarchy of research designs. N Engl J Med. 2000;342:1887–1892
  11. Juni P, Witschi A, Bloch R, et al.  The hazards of scoring the quality of clinical trials for meta-analysis. JAMA. 1999;282:1054–1060
  12. Waldholz M. A South African doctor admits falsifying cancer-treatment data. Wall Street Journal. February 7, 2000;B2

☆ Reprints not available from the author

☆☆ Address for correspondence: David L. Schriger, MD, MPH, 924 Westwood Boulevard, Suite 300, Los Angeles, CA 90024-2924; 310-794-0593, fax,310-794-0599; E-mail schriger@ucla.edu.

PII: S0196-0644(00)38560-2

doi:10.1067/mem.2000.110997

Refers to article:

  • One is the only number that you’ll ever need!

    Brian H. Rowe, Terry Klassen, Peter C. Wyer
    Annals of Emergency Medicine November 2000 (Vol. 36, Issue 5, Pages 520-523)

Annals of Emergency Medicine
Volume 36, Issue 5 , Pages 517-519, November 2000

Article Tools