Assessment of the Reliability of the Johns Hopkins/Agency for Healthcare Research and Quality Hospital Disaster Drill Evaluation Tool
Received 13 April 2007; received in revised form 24 July 2007; accepted 31 July 2007. published online 15 October 2007.
Refers to article:
Smoke and Shadows: Measuring Hospital Disaster Preparedness
Jonathan L. Burstein
Annals of Emergency Medicine
September 2008 (Vol. 52, Issue 3, Pages 230-231) Full Text |
Full-Text PDF (63 KB)
Study objective
The Joint Commission requires hospitals to implement 2 disaster drills per year to test the response phase of their emergency management plans. Despite this requirement, there is no direct evidence that such drills improve disaster response. Furthermore, there is no generally accepted, validated tool to evaluate hospital performance during disaster drills. We characterize the internal and interrater reliability of a hospital disaster drill performance evaluation tool developed by the Johns Hopkins University Evidence-based Practice Center, under contract from the Agency for Healthcare Research and Quality (AHRQ).
Methods
We evaluated the reliability of the Johns Hopkins/AHRQ drill performance evaluation tool by applying it to multiple hospitals in Los Angeles County, CA, participating in the November 2005 California statewide disaster drill. Thirty-two fourth-year medical student observers were deployed to specific zones (incident command, triage, treatment, and decontamination) in participating hospitals. Each observer completed common tool items, as well as tool items specific to their hospital zone. Two hundred items from the tool were dichotomously coded as indicating better versus poorer preparedness. An unweighted “raw performance” score was calculated by summing these dichotomous indicators. To quantify internal reliability, we calculated the Kuder-Richardson interitem consistency coefficient, and to assess interrater reliability, we computed the κ coefficient for each of the 11 pairs of observers who were deployed within the same hospital and zone.
Results
Of 17 invited hospitals, 6 agreed to participate. The raw performance scores for the 94 common items ranged from 18 (19%) to 63 (67%) across hospitals and zones. The raw performance scores of zone-specific items ranged from 14 of 45 (31%) to 30 of 45 (67%) in the incident command zone, from 2 of 17 (12%) to 15 of 17 (88%) in the triage zone, from 19 of 26 (73%) to 22 of 26 (85%) in the treatment zone, and from 2 of 18 (11%) to 10 of 18 (56%) in the decontamination zone. The Kuder-Richardson internal reliability, by zone, ranged from 0.72 (95% confidence interval [CI] 0.58 to 0.87) in the treatment zone to 0.97 (95% CI 0.95 to 0.99) in the incident command zone. The interrater reliability ranged, across hospital zones, from 0.24 (95% CI 0.09 to 0.38) to 0.72 (95% CI 0.63 to 0.81) for the 11 pairs of observers.
Conclusion
We found a high degree of internal reliability in the AHRQ instrument's items, suggesting the underlying construct of hospital preparedness is valid. Conversely, we found substantial variability in interrater reliability, suggesting that the instrument needs revision or substantial user training, as well as verification of interrater reliability in a particular setting before use.
aDepartment of Emergency Medicine, Harbor–UCLA Medical Center, Torrance, CA
bDavid Geffen School of Medicine at UCLA, Los Angeles, CA
cLos Angeles Biomedical Research Institute, Torrance, CA
dThe South Bay Disaster Resource Center at Harbor–UCLA Medical Center, Los Angeles, CA
Address for correspondence: Amy H. Kaji, MD, MPH, Department of Emergency Medicine, Harbor–UCLA Medical Center, 1000 West Carson Street, Box 21, Torrance, CA 90509; 310-222-3500, fax 310-782-1763
Supervising editor: Jonathan L. Burstein, MD
Author contributions: AHK and RJL conceived and designed the study, obtained research funding, supervised the conduct of the data collection, had full access to the data, and take full responsibility for the integrity of the data and the accuracy of the data analysis. AHK undertook recruitment of participating centers and managed the data. AHK and RJL analyzed the data. AHK drafted the article, and both authors contributed substantially to its revision. AHK takes responsibility for the paper as a whole.
Funding and support: By Annals policy, all authors are required to disclose any and all commercial, financial, and other relationships in any way related to the subject of this article, that might create any potential conflict of interest. The authors have stated that no such relationships exist. See the Manuscript Submission Agreement in this issue for examples of specific conflicts covered by this statement.
Publication dates: Available online October 15, 2007.
ABSTRACT