Interrater Reliability and Accuracy of Clinicians and Trained Research Assistants Performing Prospective Data Collection in Emergency Department Patients With Potential Acute Coronary Syndrome
Received 14 September 2008; received in revised form 4 November 2008; accepted 26 November 2008. published online 30 January 2009.
Study objective
Clinical research requires high-quality data collection. Data collected at the emergency department evaluation is generally considered more precise than data collected through chart abstraction but is cumbersome and time consuming. We test whether trained research assistants without a medical background can obtain clinical research data as accurately as physicians. We hypothesize that they would be at least as accurate because they would not be distracted by clinical requirements.
Methods
We conducted a prospective comparative study of 33 trained research assistants and 39 physicians (35 residents) to assess interrater reliability with respect to guideline-recommended clinical research data. Immediately after the research assistant and clinician evaluation, the data were compared by a tiebreaker third person who forced the patient to choose one of the 2 answers as the correct one when responses were discordant. Crude percentage agreement and interrater reliability were assessed (κ statistic).
Results
One hundred forty-three patients were recruited (mean age 50.7 years; 47% female patients). Overall, the median agreement was 81% (interquartile range [IQR] 73% to 92%) and interrater reliability was fair (κ value 0.36 [IQR 0.26 to 0.52]) but varied across categories of data: cardiac risk factors (median 86% [IQR 81% to 93%]; median 0.69 [IQR 0.62 to 0.83]), other cardiac history (median 93% [IQR 79% to 95%]; median 0.56 [IQR 0.29 to 0.77]), pain location (median 92% [IR 86% to 94%]; median 0.37 [IQR 0.25 to 0.29]), radiation (median 86% [IQR 85% to 87%]; median 0.37 [IQR 0.26 to 0.42]), quality (median 85% [IQR 75% to 94%]; median 0.29 [IQR 0.23 to 0.40]), and associated symptoms (median 74% [IQR 65% to 78%]; median 0.28 [IQR 0.20 to 0.40]). When discordant information was obtained, the research assistant was more often correct (median 64% [IQR 53% to 72%]).
Conclusion
The relatively fair interrater reliability observed in our study is consistent with previous studies evaluating interrater reliability for cardiovascular disease in the inpatient setting. With respect to research data, we found that prospective ascertainment of clinical data is more often correct when done by research assistants compared with clinicians simultaneously evaluating patients.
Department of Emergency Medicine, Hospital of the University of Pennsylvania, Philadelphia, PA
Address for correspondence: Judd E. Hollander, MD, Department of Emergency Medicine, University of Pennsylvania, Ground Floor, Ravdin Building, 3400 Spruce Street, Philadelphia, PA 19104-4283
Supervising editors: Jason S. Haukoos, MD, MS; Steven M. Green, MD
Dr. Haukoos and Dr. Green were the supervising editors on this article. Dr. Hollander did not participate in the editorial review or decision to publish this article.
Author contributions: COC, EBM, CMM, AMC, and JEH contributed to data collection. COC, FSS, AMC, and JEH wrote the article. COC, EBM, FSS, CMM, and AMC critically reviewed the article. EBM, FSS, CMM, and JEH contributed to study design. EBM, FSS, CMM, AMC, and JEH analyzed and interpreted the data. FSS was overall statistician for the project. JEH was the principal investigator for the study, collated comments from other authors, and takes overall responsibility for the data. COC takes responsibility for the paper as a whole.
Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article that might create any potential conflict of interest. The authors have stated that no such relationships exist. See the Manuscript Submission Agreement in this issue for examples of specific conflicts covered by this statement.
Publication date: Available online January 29, 2009.
ABSTRACT
Interrater Reliability and Accuracy of Clinicians and Trained Research Assistants Performing Prospective Data Collection in Emergency Department Patients With Potential Acute Coronary Syndrome