Medical Record Reviews in Emergency Medicine: The Blessing and the Curse

The clinical record freezes in time that episode in life called illness. It is a story in which patient and family are the main characters, with the doctor serving a dual purpose as both biographer and part of the plot.1

More than 20 years ago, my colleagues and I conducted our first medical record review. We hypothesized that alcohol intoxication was a common cause of acute atrial fibrillation in emergency department (ED) patients. To find out if this was so, we reviewed 40 ED records. We included “all the charts we could find,” not thinking much about patient selection. We did not define key variables. The author-abstractors were untrained, unmonitored, and unblinded to the alcohol status of the patients. We had never heard of interrater reliability and did not measure it. Our medical record review was published in the Archives of Internal Medicine in 1983.2 It would not be published anywhere today.

Today, there are published standards for conducting and reporting medical record reviews.3, 4, 5 At a minimum, every published medical record review should provide information in 7 key methodologic areas: (1) explicit protocols for case selection and exclusion; (2) abstractor training; (3) precise definitions of key variables; (4) use of standardized abstraction and coding forms; (5) monitoring of abstractor performance; (6) blinding of abstractors to study hypotheses and patient groups; and (7) testing of interrater agreement. The first standard is essential so that readers may understand the sampling methodology, source population, and eligibility and exclusion criteria. Standards 2 to 5 establish a priori protocols for data abstraction, in order to increase the validity and reproducibility of the data. Standard 6 counters the bias that results from chart abstractors' hunches and hopes regarding outcomes and associations. The last standard provides a measure of the reproducibility of the data. If information about these variables is missing, the conclusions of the medical record review may not be credible.

In 1996, we reviewed the status of medical record reviews in the 3 leading emergency medicine journals.3 During the 5-year period ending December 31, 1993, 986 original scientific investigations were published; of these, 244 (25%) were medical record reviews. Adherence to the 7 methodologic standards varied. While 98% of the medical record reviews described inclusion and exclusion criteria, most articles did not report abstractor training (18%), use of standardized abstraction forms (11%), monitoring of abstractors (4%), blinding of abstractors (3%), or testing of interrater reliability (0.4%). We asked where the methods are in emergency medicine medical record reviews, and we suggested that adherence to the 7 standards would improve the quality, clarity, and trustworthiness of these investigations.

Now, a decade later, it is appropriate to take stock and inquire: With respect to published medical record reviews, are we better off today than we were 10 years ago? The answer is “No,” according to 2 articles published in the current issue of Annals. Worster et al6 found that, in 2003, medical record reviews accounted for 8% to 29% of all scientific publications. There was significant improvement in rigor in 2 areas: data abstraction forms and reporting of interrater reliability. However, even in 2003, interrater reliability was tested in just 19% of all published articles. Only 4% reported abstractor blinding, which is the same percentage as in 1993. Badcock et al7 reported improvement in 4 areas: abstractor performance monitoring, use of standardized abstraction forms, blinding, and attention paid to interrater reliability. But even in 2000 to 2004, details about blinding and interrater reliability were reported in only 7% and 28% of medical record reviews, respectively. In both reports, adherence to these 7 standards varied markedly across the different peer-reviewed journals and remained suboptimal overall.

The studies by Worster et al6 and Badcock et al7 are noteworthy in 3 ways. First, Badcock et al's report included the Emergency Medicine Journal (published in the United Kingdom) and Emergency Medicine Australasia (from Australia). Now we know that the absence of methodologic rigor in emergency medicine medical record reviews is an international problem. Second is the disappointing finding that Annals of Emergency Medicine, which lists these standards as requirements in its “Instructions for Authors,” fared only marginally better than the other peer-reviewed journals.5 Third is the reassuring finding that Worster et al and Badcock et al chose to eat their own cooking: in each of these articles, the authors took the time to provide details about abstractor training, monitoring, blinding, exclusion criteria, key variables, resolution of coding conflicts, and their own interrater reliability.6, 7

Badcock et al7 and Worster et al6 also suggest that medical record reviews should adhere to several additional methodologic standards. According to Badcock et al, medical record reviews should include a sample size calculation, evidence of human subjects review board approval, and a disclosure of all funding sources. These are important elements that are typically required for any submission to a medical journal. Worster et al add that medical record reviews should include a clear description of the sampling methodology (convenience, consecutive, systematic, or random), plus a discussion of missing charts and the strategies used to manage missing data. These are worthy additions to the list.

Who would argue with the principal conclusions of these 2 articles—namely, that adherence to the methodologic standards for medical record reviews remains suboptimal? After a decade, medical record reviews are still governed by the “laws of laissez faire: the investigator usually chooses the records and removes the data in whatever manner he wishes, and he seldom reports the details of this method.”8 In the simplest terms, there are gaps in the quality control procedures for manuscript preparation, review, and publication. Strict adherence to the standards first recommended in 1993, and now again by Worster et al6 and Badcock et al,7 would help authors, journal editors, reviewers, and readers “in their mutual task of creating and distributing accurate, clear, easily accessible reports of biomedical studies.”9

In the remainder of this editorial, I will address 3 related questions regarding medical record reviews in the emergency medicine literature: (1) In an era ruled by evidence-based medicine, do medical record reviews still have value?; (2) What are the key steps that an investigator should take when considering embarking on a medical record review?; and (3) Will medical record reviews become obsolete in the era of electronic medical records?

Do Medical Record Reviews Have Value? The Blessing and the Curse 

Medical record reviews are research studies in which the data of interest derive from clinical records. The data were recorded by doctors, nurses, paramedics, and other providers during the course of ordinary medical care. This definition excludes systematic or administrative databases, such as state trauma registries or the National Hospital Ambulatory Medical Care Survey (NHAMCS). The information in medical records is always recorded at the point of care, whereas the research is planned and conducted weeks, months, or years later. And therein lies the blessing and the curse.

The Blessing 

Medical record reviews are clumsy and imprecise. But as the only written record of the illness experience, they have “prima facia value.”10 Medical record reviews are a source of patient-oriented and practice-oriented data that can provide valuable insights in several important areas: (1) disease descriptions, including patient and illness characteristics, natural history, laboratory features, and diagnostic test performance; (2) clinical-pathologic correlations; (3) quality audits, including assessments of variations in care or adherence to clinical guidelines; (4) derivation and validation of clinical prediction rules; and (5) analyses of the costs of care. Medical record reviews can seldom provide definitive answers about “best treatments,” but they can suggest hypotheses about treatment, prognostic factors, and complications that can be tested in prospective studies.11

The most important strength of medical record reviews is that they provide information about patients' experiences during ordinary medical care. Absent are the hype, heightened expectations, and sometimes skewed results of the well-funded and highly structured randomized clinical trial. Medical record reviews also provide an opportunity for clinician-scholars to use already-existing data to conduct practice-based, clinically relevant research. Almost a century ago, J. Madison Taylor observed, in the Monthly Cyclopedia of Practical Medicine, “There is an impression broadly prevalent … that it is not worthwhile for active practitioners, untrained in literary, laboratory or hospital methods, to publish their experiences … [But in fact, a manuscript based on clinical experience] may furnish excellent data from which others may deduce generalizations leading to the establishment of truths.”12

The Curse 

The principal limitation of medical record reviews is that there was no protocol for patient care. There was no systematic plan for recording chief complaints, defining diseases, conducting diagnostic workups, administering treatments, or assessing outcomes. What do “cough, mostly nonproductive,” “no signs of joint involvement,” and “patient improved, ready for discharge” really mean? No one can know.

The second limitation is that there was no protocol for committing any of the above to paper. The data are recorded in a casual manner by an army of providers and observers. The records contain an admixture of numbers and results, along with hunches and speculations about symptoms, physical findings, diagnoses, and outcomes. Data entries are often illegible, conflicting, missing, or otherwise uncodable. The fact is, for all the key elements in a medical record—histories, physical findings, test results, etiologies, exposures, severity scales and scores, discharge diagnoses, and outcomes—clinicians differ in how they observe, measure, interpret, and report their findings.3 In one study of ambulatory care, 27% of charts contained no chief complaint.13 Psychiatric complaints and illnesses may be omitted in one third of records of clinical encounters.14 Schwartz et al15 found that emergency physicians, in completing their records on trauma patients, lose one third of the available data about cause of injury when they obtain their histories; they lose an additional one third because they never write it down. When medical records are compared with videotaped or directly observed medical encounters, the written records are often incorrect.16 According to Reiser,17 “The large record typical of contemporary medical practice is filled with undigested details of laboratory, progress and nursing notes that present a daunting challenge to [investigators] seeking to discern trends or gain a comprehensive view of events.”

Errors, omissions, and idiosyncrasies of medical records are then compounded during the chart abstraction process. Finding and interpreting chart entries, coding data, and creating ordinal response categories all require training, practice, and judgment. Therefore, the abstraction process requires agreed-to protocols, training, performance monitoring, and performance review. The absence of a priori rules for data abstraction and coding also means that the medical record review is ripe for bias, not only in the selection of charts to be included, but also in the abstraction process. In a medical record review, it is easy to find just the patients, or just the chart entries, that feature the association or outcome of interest.

Even when they are transcribed, typewritten medical records may contain information that is inaccurate, incomplete, and uncodable, along with intriguing spoonerisms, malapropisms, and dangling participles. “The patient is alert but confused,” “Occasionally responsive to verbal pain,” “Patient has two teenage children but no other abnormalities,” and “We will instruct him to watch his prostate” are among dictated notes that defy coding, scaling, or analysis.

That is the curse. Hampered by uninterpretable medical records and lacking protocols for systematic chart abstraction, the investigator approaches the medical record review at a disadvantage, unable to find a clear answer to any of the “legendary cub reporter's questions.”10Who were the subjects? What was done? Who did these things? When, how often, and why?

Medical Record Reviews: What should the Investigator do? 

If medical record reviews are unreliable and prone to bias and sloth, what is the investigator to do? The answer is to recognize the limitations and challenges of the clinical record and to take steps to add rigor to the data collection process. In general, before embarking on a medical record review, the investigator should take 4 steps:

•Consider first whether there is a larger, more representative source of data. Numerous population-based data sets exist, including NHAMCS, the Fatality Analysis Recording System, the National Electronic Injury Surveillance System, and many others. Why use a medical record review to study atrial fibrillation in 40 patients in your own ED when you can use NHAMCS to make inferences about 4 million visits in EDs across the country?

•If you choose to conduct a medical record review, do not overlook the Methods section. Adhere to the methodologic reporting standards (as outlined by Gilbert et al,3 Badcock et al,7 and Worster et al6) so that reviewers and readers are not left in the dark about the applicability, validity, and reliability of the results. The patient care and recording may have taken place without a protocol, but the medical record review should have one.

•Be cautious about the conclusions you draw from a medical record review, especially about differences in outcomes in subsets of patients. None of the subsets were randomized, and the diagnostic assessments and medical interventions were not standardized. “One can always expect to find some good-looking and some bad-looking subsets in any body of data.”10 All the more so in a series of patient charts.

•Remember that adherence to the reporting standards does not mean, necessarily, that the medical record review should be published. The discerning reader or journal editor will also want to know whether the medical record review is newsworthy. How does the report stand up in terms of ingenuity, novelty, and relevance to science or patient care? The 18th-century novelist (and 2-time journal editor) Henry Fielding once wrote, “Such histories as these do, in reality, much resemble a daily newspaper, which consists of just the same number of words, whether there be any news in it or not.”18

The Era of Electronic Medical Records: What Lies Ahead for Medical Record Reviews? 

In the very near future, the bulky, scrappy folders that we call medical charts will probably disappear. There will only be electronic medical records, with check-box entries and computer-driven choice fields. Will this eliminate the “curse” of the medical record review? Perhaps not. The records will be easier to locate and review. Vast amounts of automated data will be retrievable, at a lower cost. All the entries will be legible. At the same time, medical records will still be prone to inaccurate and conflicting entries and missing data. In addition, as Burnum,19 Kassirer,20 Reiser,17 and others have noted, medical records are likely to undergo new distortions, as physicians completing these records increasingly succumb to outside pressures such as privacy concerns, billing imperatives, and worries about physician profilers, utilization auditors, and malpractice lawyers. Electronic medical records may become populated with “incomplete complaints” and truncated narratives about the course, outcomes, and complications of medical care, rendering them less authentic.17, 18, 19 The brief keystrokes of the computer will also misrepresent or conceal more and more of that “episode in life called illness,” making these records less lively and less interesting for the clinician-investigator to read.