Is Evidence-based Medicine a Teachable Skill?☆☆☆
Article Outline
Abstract
[Hatala R: Is evidence-based medicine a teachable skill? Ann Emerg Med August 1999;34:226-228.]
See related article, p. 148 .
As we all remember from our undergraduate and postgraduate training, learning new material can be difficult. It may require multiple repetitions through the material, more than one modality of instruction, and motivation on the part of the learner. Otherwise, we may have gained sufficient knowledge to pass a course test, but our true understanding of the material and our ability to apply it to clinical practice would be in doubt.
In this issue of the Annals of Emergency Medicine , Bazarian et al1 report the results of an attempt to teach a component of evidence-based medicine (EBM), critical appraisal, to emergency medicine residents. They are not the first to have encountered difficulties in translating the knowledge and skills they wished the residents to obtain into demonstrable gains in knowledge. Their study highlights some of the difficulties in attempting to study the efficacy of educational interventions in general, and in the domain of EBM curricula specifically.
What makes research into educational interventionsso difficult? First, our understanding of the cognitiveprocesses involved in learning medicine is limited.2 Interventions that appear innovative may not reflect a new innovation from the perspective of the psychological process involved in the learning. Second, an individual’s ability to learn is influenced by a myriad of factors including the motivation of the learner and the instructor, the time spent on the learning task, and the learning environment. Third, even when an effective educational intervention is devised, studies tend to include small samples (usually the number of students at an educational institution) and weak outcome measures that limit their ability to detect a benefit.
EBM describes an approach to clinical decisionmaking that integrates “individual clinical expertise with the best available external evidence from systematic research.”3 Despite widespread recognition that EBM is a useful approach to clinical decisionmaking, surprisingly few studies have demonstrated successful teaching of EBM skills to junior learners.4 EBM practice involves a complex set of knowledge, skills, and attitudes. A competent practitioner of EBM has the ability to articulate a focused clinical question based on a clinical encounter with a patient and has sufficient knowledge of the available resources to efficiently search the relevant literature and select the best available evidence. The competent practitioner is able to critically appraise the retrieved literature, assessing both the quality of the research and the magnitude of the results. Most importantly, the EBM practitioner decides with the individual patient whether to integrate the results of the literature in the patient’s clinical care.5
Recognizing that the assessment of these behaviors is complicated, most educational studies have focused on teaching critical appraisal skills instead of tackling the broader issue of teaching EBM. Even for a skill as clearly defined as critical appraisal, there have been difficulties objectively assessing students’ learning gains. Critical appraisal skills build on general knowledge of clinical epidemiology. Multiple-choice tests, the most common form of assessment used in previous studies, tend to test student’s knowledge of epidemiology and biostatistics rather than their critical appraisal skills. For example, a multiple-choice question focused on clinical epidemiology might ask students which study design would best assess a particular research question. However, to test critical appraisal skills, the student would need to assess a specific study, analyzing its strengths and weaknesses. Striving to measure critical appraisal skills, a few studies required the learner to critique an article but only 1 of these studies6 adequately assessed the reliability and validity of their measurement tool.6, 7, 8
A comprehensive review of this literature on teaching critical appraisal reveals that medical students can improve their critical appraisal skills after an educational intervention. However, journal club formats of critical appraisal instruction to residents yield little demonstrable knowledge gains.4 The limitations in the assessment tools used in these studies make difficult the interpretation of their results. Learners may have gained important skills that are not captured by the assessment measures. In addition, most of these studies used a single educational intervention, such as a journal club or lecture series, to teach critical appraisal. Considering the numerous educational demands on house staff, single educational interventions are unlikely to have a significant effect on learning.
In an important extension of the previous studies, Bazarian et al1 developed a multifaceted approach to teaching critical appraisal. The authors’ instructional approach included one-on-one faculty time with the house staff, followed by the learners teaching the material to their peers. The lack of demonstrable benefit to the educational intervention may reflect a problem with their method of assessment rather than a limitation in the instructional approach. In their study, the same test (analysis of a methodologically flawed article) was administered to the residents both before and after the educational intervention. Rather than assessing whether the residents learned critical appraisal skills, their study inadvertently assessed whether the residents became better at taking the test.
By way of contrast, a study that involved a multifaceted intervention similar to Bazarian et al’s, was able to demonstrate significant gains in residents’ EBM behaviors, self-confidence, and skills in critical appraisal.9 In addition to one-on-one mentoring and peer tutoring, this study (1) undertook a needs assessment before the intervention to ensure residents were interested in learning EBM, and (2) included faculty development in the intervention. Considered with Bazarian et al’s intervention, these studies underscore the importance of developing multifaceted approaches to education if we intend our learners to make lasting knowledge gains.
Is EBM a teachable skill? This question cannot be answered conclusively from the literature to date. We know that EBM requires a specialized set of knowledge, skills, and attitudes before it can be integrated into clinical care. Single educational interventions, such as a journal club or lecture series, are unlikely to make a significant effect on house staff’s learning. Local culture as reflected in the attitudes of learners and faculty are key to any curricular change. As one of the first efforts to develop an EBM curriculum, the McMaster Internal Medicine Residency Training program emphasized the importance of recruiting and developing clinical faculty as enthusiastic and skilled EBM role models for resident learning.5 Ultimately, there may not be an effective shortcut to such a far-reaching approach. As in much of educational research, if we are to demonstrate the effectiveness of different approaches to EBM teaching, we must strive for better assessment tools. These assessment tools will help us examine whether the next generation of physicians have successfully integrated EBM in their approach to clinical care.
References
- Teaching emergency medicine residents evidence-based critical appraisal skills: A controlled trial. Ann Emerg Med. 1999;34:148–154
- . Mental representations of medical diagnostic knowledge: A review. Acad Med. 1996;71(Suppl):S55–S61
- Evidence based medicine: What it is and what it isn’t. BMJ. 1996;312:71–72
- . Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: A critical appraisal. Can Med Assoc J. 1998;158:177–181
- . Evidence-Based Medicine: A new approach to teaching the practice of medicine. JAMA. 1992;268:2420–2425
- A controlled trial of teaching critical appraisal of the clinical literature to medical students. JAMA. 1987;257:2451–2454
- Impact of a medical journal club on house staff reading habits, knowledge and critical appraisal: A randomized, controlled trial. JAMA. 1988;260:2537–2541
- Evaluating medical residents’ literature-appraisal skills. Acad Med. 1995;70:152–154
- . Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997;12:742–750
☆ Reprints not available from the author.
☆☆ 47/1/99872
PII: S0196-0644(99)70236-2
© 1999 American College of Emergency Physicians. Published by Elsevier Inc. All rights reserved.
