Interrater Reliability and Accuracy of Clinicians and Trained Research Assistants Performing Prospective Data Collection in Emergency Department Patients With Potential Acute Coronary Syndrome

Introduction 

Materials and Methods 

Results 

There were 143 patients recruited. Forty-seven percent were female patients, and they had a median age of 51 years (SD 13). The interviews were performed by 33 research assistants (58% women) and 39 clinicians (59% women). The clinicians consisted of 19 emergency medicine residents, 16 non–emergency medicine residents, and 4 emergency medicine attending physicians.

Overall, across all individual items assessed (Table 1), the median agreement and interrater reliability were both fair (81% crude agreement [IQR 73% to 92%]; κ value 0.36 [IQR 0.26 to 0.52]). With regard to specific categories (Table 2), agreement was good for traditional cardiac risk factors (hypertension, diabetes, hypercholesterolemia, smoking, family history of myocardial infarction, and cocaine use) and was moderate for other cardiac history. There was good to excellent agreement, however, for certain components of the cardiac history (congestive heart failure, coronary artery disease, myocardial infarction, and coronary artery bypass grafting). There was moderate agreement for the presence and results of previous cardiac testing. The agreement was fair for pain location, radiation, quality, and associated symptoms. The general chest pain questions not falling into any of the other defined categories also had fair agreement. The discordance could not be accounted for by individual patients because the majority of patients had at least 1 variable that was discordant.

Table 1. Reliability of clinical histories between clinicians and research assistants.

	Characteristics	κ	95% CI	Crude Agreement, %	95% CI
	General chest pain questions
	Chest pain chief complaint?	0.37	0.19 to 0.55	78	70 to 84
	Chest pain at rest?	0.21	0.04 to 0.38	72	64 to 79
	Multiple chest pain episodes?	0.33	0.17 to 0.49	67	59 to 75
	Chest pain on ED arrival?	0.28	0.11 to 0.44	76	68 to 83
	Worse than previous angina?	0.31	0.15 to 0.47	71	63 to 79
	Chest pain pleuritic?	0.29	0.14 to 0.44	65	57 to 73
	Chest pain quality
	None	0.39	−0.15 to 0.93	98	94 to 100
	Crushing	0.07	−0.11 to 0.25	83	76 to 89
	Pressure/tightness	0.30	0.15 to 0.46	66	58 to 74
	Aching/dull	0.42	0.25 to 0.59	78	70 to 84
	Burning	0.14	−0.07 to 0.35	87	81 to 92
	Tearing	0.27	−0.18 to 0.71	97	92 to 99
	Sharp/stabbing	0.26	0.10 to 0.43	68	60 to 75
	Other/unknown	0.44	0.14 to 0.74	94	88 to 97
	Chest pain location
	None	⁎	⁎	97	92 to 99
	Substernal/midchest	0.47	0.32 to 0.61	73	65 to 80
	Left anterior chest	0.52	0.38 to 0.66	76	68 to 83
	Left arm	0.37	0.05 to 0.69	94	88 to 97
	Right anterior chest	0.58	0.37 to 0.79	92	86 to 96
	Epigastric	0.30	0.05 to 0.56	90	83 to 94
	Other	0.21	−0.08 to 0.50	92	86 to 96
	Unknown	−0.01	−0.02 to 0.00	99	95 to 100
	Chest pain radiation
	None	0.35	0.21 to 0.49	67	59 to 75
	Left arm	0.57	0.40 to 0.74	86	79 to 91
	Jaw/neck	0.38	0.17 to 0.59	86	79 to 91
	Back	0.23	−0.01 to 0.46	87	80 to 92
	Right arm	0.44	0.09 to 0.78	95	90 to 98
	Other	−0.05	−0.09 to (−0.01)	85	78 to 90
	Associated symptoms
	Shortness of breath	0.44	0.30 to 0.59	73	65 to 80
	Diaphoresis	0.31	0.14 to 0.48	77	69 to 84
	Nausea	0.42	0.28 to 0.56	74	66 to 81
	Vomiting	0.40	0.16 to 0.64	90	83 to 94
	Light headed	0.25	0.14 to 0.35	57	49 to 66
	Syncope	0.11	−0.03 to 0.24	81	74 to 87
	Palpitations	0.23	0.09 to 0.37	68	59 to 75
	Weakness	0.09	0.00 to 0.19	48	40 to 57
	Cardiac risk factors
	Hypertension	0.75	0.63 to 0.86	88	82 to 93
	Diabetes mellitus	0.86	0.76 to 0.95	94	89 to 98
	Increased cholesterol level	0.63	0.49 to 0.76	84	77 to 90
	Family history of myocardial infarction (<55 y)	0.35	0.20 to 0.50	73	64 to 80
	Tobacco use (now or past)	0.61	0.49 to 0.74	80	73 to 87
	Cocaine use (<1 week ago)	0.92	0.76 to 1.00	99	96 to 100
	Other cardiac history
	Congestive heart failure	0.75	0.60 to 0.90	93	88 to 97
	Coronary artery disease	0.77	0.64 to 0.91	93	88 to 97
	Angina	0.35	0.17 to 0.54	79	71 to 85
	Myocardial infarction	0.90	0.80 to 1.00	97	93 to 99
	Undiagnosed chest pain	0.14	−0.02 to 0.30	73	65 to 80
	GERD/peptic ulcer disease	0.26	0.10 to 0.43	73	65 to 80
	Arrhythmia	0.29	0.11 to 0.46	80	73 to 87
	Coronary artery bypass grafting	0.88	0.72 to 1.00	99	95 to 100
	Known cancer	0.56	0.27 to 0.85	95	90 to 98
	Cardiac testing
	Exercise stress test	0.52	0.39 to 0.65	75	66 to 81
	Exercise stress test result	0.24	0.03 to 0.45	53	38 to 67
	Cardiac catheterization	0.64	0.49 to 0.79	87	79 to 91
	Cardiac catheterization result	0.46	0.16 to 0.76	67	45 to 84

CI, Confidence interval; GERD, gastroesophageal disease. Unweighted κ scores for all items.

⁎ A κ value was impossible to determine for “Chest pain location: None” because no clinicians answered this question in the affirmative.

Table 2. Reliability of clinical histories between clinicians and research assistants: Median κ values and crude agreement across question categories.

	Category of Questions	κ	IQR	Crude Agreement, %	IQR, %
	General questions	0.30	0.28-0.32	72	68-75
	Chest pain quality	0.29	0.23-0.40	85	75-94
	Chest pain location	0.37	0.25-0.49	92	86-94
	Chest pain radiation	0.37	0.26-0.42	86	85-87
	Associated symptoms	0.28	0.20-0.40	74	65-78
	Cardiac risk factors	0.69	0.62-0.83	86	81-93
	Other cardiac history	0.56	0.29-0.77	93	79-95
	Cardiac testing	0.49	0.41-0.55	71	63-78

Unweighted κ scores for all items.

When there were discordances in data collection, the research assistant was more often correct than the clinician (64% [IQR 53% to 72%]) (Table 3). Furthermore, in half the categories, the research assistants obtained the accurate information more than 70% of the time.

Table 3. Accuracy of research assistant–collected data in cases in which it differed from that of the clinicians.

	Category of Questions	% Research Assistant Correct	IQR, %
	General questions	53.0	46-59
	Chest pain quality	71.0	68-82
	Chest pain location	55.0	44-64
	Chest pain radiation	66.0	57-68
	Associated symptoms	71.0	64-76
	Cardiac risk factors	77.0	60-85
	Other cardiac history	60.0	50-67
	Cardiac testing	76.0	66-87

Limitations 

This study can be criticized for using a third “tiebreaker” who was not an experienced clinician. We chose the study design for several reasons. First, we were comparing the concordance of the treating clinician to the research assistant. We could have had the attending physician history serve as the tiebreaker, but we believed that a more directed approach, whereby the tiebreaker could directly present the patient with the 2 previous choices and ask him or her to choose between them, was more appropriate. We were concerned that the attending physician would have already evaluated the patient, possibly with the clinician in the room, and the patient would therefore feel compelled to answer in a manner similar to whatever they told the medical team. Direct confrontation by a third person who had not previously evaluated the patient was chosen to avoid that source of bias.

The use of a convenience sample may have created selection bias, but we attempted to limit this by screening all patients with chest pain when research assistants were available. We did not control for the clinicians' or research assistants' level of training. Previous studies suggest that there is a higher concordance among observers with similar training.18, 19 Our research assistants were largely premedical post-baccalaureate students, who may have more medical interest or expertise than other research assistants. We did not control for whether the clinician or research assistant talked with the patient first. Patients presenting with a disorganized list of complaints may sharpen their focus with subsequent interviews.20 This “coaching” effect may have inflated the degree of concordance. It is also possible that the research assistants, who focused only on the study-related data, were likely to overcome this problem. It is also likely that the research assistants used the structured form while interviewing the patient and the resident did not complete the structured form until after the patient assessment; however, we did not directly assess this and cannot be sure whether this occurred.

It is difficult to quantify how much patient-related factors decreased the interrater reliability for each category of questions. As mentioned previously, patients often have poor or nonspecific recall that can vary with time. Moreover, the ability to interpret questions and provide coherent answers is generally more impaired in patients who are old, cognitively impaired, less medically knowledgeable, and not native English speakers.20 These problems may be further compounded by a lack of honesty by patients as a result of mistrust or dissatisfaction with their care.20 Although strong clinical and interpersonal skills should help the clinician mitigate these difficulties, it is improbable that these difficulties are completely overcome routinely.

There are several well-described limitations to the use of the κ statistic that influence the interpretation of the results.21, 22 When there is a very high prevalence of affirmative or negative answers, the level of expected agreement becomes so great that κ becomes difficult to interpret.21 For this reason, it is important to inspect the raw data for prevalence effects that artificially lower κ values. Statistical methods to adjust for prevalence have been proposed, and some experts believe that the intraclass correlation coefficient is a better measure, but nevertheless, κ remains the most widely used index of agreement.23

Finally, our sample size was chosen on the basis of convenience rather than a priori statistical calculations. The IQR for categories of questions is up to 15 percentage points wide. Nonetheless, our percentage of accuracy favoring the trained research assistants supports our conclusion that they are at least as accurate as the clinicians, in general.

Discussion 

The relatively fair interrater reliability observed in our study is consistent with expected results based on previous studies evaluating interrater reliability for cardiovascular disease in the inpatient setting. To our knowledge, the only study that directly examined physician-physician observer variability in patients with chest pain was performed by Hickam et al.18 They assessed the interobserver agreement between 2 general internists in a study of 197 inpatients admitted with chest pain. The 2 internists had overall good agreement when asking similar questions to the ones found on our clinical data form. κ Values for pain radiating to the left arm, history of myocardial infarction, pain in substernal location, pain described as pressure, pain brought on by cough or deep breath, and pain described as sharp were 0.89, 0.78, 0.74, 0.57, 0.44, and 0.30, respectively. Although their degree of agreement was higher than that found in our study, it was likely artificially inflated because the internists began the study by discussing how to interpret hypothetical responses and how to make it so that they approach the patients similarly. They also approached patients in a less hectic setting after the initial management decisions had already been made and the patients had already been questioned by the clinicians providing their medical care.

In contrast, our study demonstrated agreement for cardiac risk factors that was equal to or higher than that in a study by Hand et al.19 This study examined the interrater reliability among 3 physicians and a medical student who interviewed 98 patients presenting with potential stroke. Although a different patient population than our own, patients presenting with potential stroke also require inquiry about cardiac risk factors as an aid to diagnosis. The Hand study's κ values for a history of smoking, diabetes mellitus, ischemic heart disease, atrial fibrillation, and hypertension were 0.69, 0.65, 0.64, 0.54, and 0.47, respectively.

We observed several trends that paralleled those of previous chest pain studies. There was greater agreement for classic angina symptoms such as shortness of breath, nausea, vomiting, and diaphoresis than for atypical or nonspecific symptoms.1 Indeed, nonspecific symptoms are often overlooked, possibly as a result of their being attributed to less severe illnesses.23, 24 We also found that different observers were more likely to agree with regard to the location and radiation of chest pain than with regard to its quality,18 which makes intuitive sense because patients frequently have trouble describing chest pain whether they use their own words or those suggested by a physician.20

In reference to our study's ultimate goal, we showed that the research assistants obtained the accurate information more often than the clinicians. We expected these results because, although less experienced, research assistants do not have to deal with the clinicians' competing clinical responsibilities. The research assistants do not have to conform to the same time requirements, and they can also focus entirely on asking each of the data collection form's questions explicitly.

In conclusion, we found that when clinical histories were recorded, the agreement between non–medically trained research assistants and clinicians is comparable to the agreement observed between physicians in previous chest pain studies. More important, we found that the research assistants were more likely than the clinicians to acquire the accurate data, as assessed by our tiebreaker. We thus conclude that it is appropriate for research assistants to prospectively record clinical histories in studies of patients presenting to the ED with chest pain. This utilization of research assistants should help expand limited labor resources and thus afford physicians the opportunity to manage additional clinical and research duties.