Annals of Emergency Medicine
Volume 37, Issue 6 , Pages 629-632, June 2001

Appropriate standards for “appropriateness” research☆

Department of Biostatistics and Epidemiology Department of Emergency Medicine Center for Clinical Epidemiology and Biostatistics Leonard Davis Institute for Health Economics University of Pennsylvania Medical Center Philadelphia, PA Department of Emergency Medicine University of Pennsylvania Medical Center Philadelphia, PA

Article Outline

Abstract 

[Lowe RA, Abbuhl SB. Appropriate standards for “appropriateness” research. Ann Emerg Med. June 2001;37:629-632.]

 

See related articles, p. 568 and p. 580.

Two articles1, 2 in this month’s issue of Annals address the “appropriatenes” of emergency department visits. What makes these articles exciting is that they extend previous research on this topic in the United States to 2 European countries, Spain1 and Portugal.2 There are many reasons to expect that ED use in other countries would differ from that in the United States. The presence of universal health insurance in both Spain and Portugal, along with other differences in disease patterns, medical care delivery systems, and social structures, could lead to ED patient populations dramatically different from those seen in most US hospitals.

As US physicians who have not had the benefit of working in Europe, we will not presume to comment on the policy implications of these studies in Spain1 and Portugal.2 However, these articles are potentially important to domestic readers of Annals because they complement the large body of literature on appropriateness of ED visits in the United States. Like most US articles on this topic, the Spanish and Portuguese articles attempt to measure the proportion of ED visits that are “inappropriate.” Then, like many US studies, they report characteristics of patients using the ED inappropriately. In our view, most of the US literature on inappropriate ED visits suffers from a series of fundamental flaws. Therefore, we wish to suggest potential standards for research on appropriateness of ED use, drawing from the principles of evidence-based medicine, as well as from our own experience with this topic.3, 4, 5, 6, 7 We propose 5 questions for evaluating articles on appropriateness of ED use.

Back to Article Outline

How is appropriateness defined? 

Studies of ED use in the United States, most of which use subjective definitions of appropriateness,8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 report anywhere from 11%13 to 82%14 of ED visits as inappropriate or nonurgent. Although some of this variation represents real differences among study sites with respect to case mix, it is probable that much of the variation is because of differences in how visits are classified. There are many possible ways to assess the appropriateness of an ED visit. Some researchers have relied on the triage nurse’s assessment. This assessment can be subjective, either as a dichotomous variable (emergency versus nonemergency) or as a triage score without formal guidelines. Alternatively, the nurse’s assessment can be structured by using standardized criteria for triage scores22, 23 or other standardized criteria for classifying the need for emergency medical care.24, 25, 26, 27

Some researchers have classified appropriateness on the basis of physicians’ determinations. In some studies, researchers have inferred that if the treating physician ordered certain diagnostic tests, performed certain treatments, or hospitalized the patient, the treating physician likely believed that the patient was in need of emergency medical care.3 Other researchers have used less structured physician assessments, such as asking the treating physician (or a physician reviewing the chart retrospectively) whether the patient’s outcome might have been worse had care been delayed.

Finally, some researchers have classified appropriateness on the basis of patients’ self-assessment of whether their problems were truly emergencies or whether patients would be willing to wait a day if they had access to a prompt appointment.3

Given the many ways in which appropriateness can be measured, what is the best measure of appropriateness? This is referred to as a “criterion standard” or “gold standard.”28 A criterion standard should have the properties of reliability and validity. The standard is reliable if multiple observers, applying the same standard, will reach the same conclusion. The standard is valid if application of the standard leads to conclusions that make sense and that agree with other generally accepted standards.

Unfortunately, no criterion standard exists for the appropriateness of ED visits. Physicians disagree on which ED visits are necessary.29 Nurses’ assessments of urgency correlate poorly with physicians’ decisions about need for hospitalization.30 Indeed, several studies have shown poor correlation between any 2 measures of appropriateness and wide variability in the proportion of ED visits that are deemed appropriate by different measures.6, 31 It is incumbent on researchers to demonstrate the reliability and validity of their measures of appropriateness.

Back to Article Outline

Do the authors address the societal context for appropriateness? 

The need for emergency medical services depends on availability of services elsewhere. Just as an ED visit for an acutely suicidal patient might be deemed inappropriate were there a psychiatric crisis center next door, an ED visit for a sore throat may be appropriate for a patient with no alternative source of care. Yet one national study showed that only 8% of traditional primary care practices would be accessible to a Medicaid enrollee who needed a prompt appointment in the evening (for fear of taking time off work) and who was unable to pay in cash.32 If a reasonable person would seek care for a particular medical problem, and if the patient has no place to go other than the ED, can we really categorize use of the only remaining safety net as inappropriate? Researchers should define the alternative sources of medical care before judging an ED visit inappropriate. Alternatively, they should make clear that they are studying the effect of poor access to care on ED use, so that readers do not confuse patients’ bad judgment with good judgment in the context of poor access.

Back to Article Outline

If the article describes characteristics of inappropriate ED visits, does it use a meaningful comparison group? 

The literature often reports characteristics of inappropriate ED visits in misleading ways. For example, authors may report a high proportion of inappropriate ED visits by young men, or they may report that among ED patients, those on Medicaid were more likely to use the ED inappropriately. Both of these statements might lead readers to erroneous conclusions because of problems with the comparison group. In the first case, no comparison group is used. In the second case, inappropriate ED users are compared with appropriate ED users, but this study design cannot answer the question of what leads patients to seek care in the ED. To answer that question, we would need to compare patients with minor problems who use the ED with patients with similar problems who seek care elsewhere. Few studies have taken this approach. The one study33 of which we are aware that used a population-based approach with nationally representative data to look at nonurgent ED visits found that 54% of persons making nonurgent ED visits were in middle or upper income groups, 69% were white, and only 21% had Medicaid or other public insurance for the poor.

Back to Article Outline

If the article describes characteristics of inappropriate ED visits, what characteristics did the authors choose to study? 

In studying variables associated with inappropriate ED visits, the questions that researchers choose to ask determine, in large part, the policy implications of their answers. Some studies have looked at patient characteristics, such as chief complaints, diagnoses, age, sex, and ethnicity. These studies reach conclusions about characteristics of so-called ED abusers that have led some policymakers to adopt gatekeeping and related strategies to reduce access to EDs. Other studies have looked at patients’ access to traditional sources of primary care by studying insurance status or self-reported access to care. These studies lead to conclusions about the need for the ED as a safety net for patients without access to care elsewhere, and such findings have led to attempts to preserve unrestricted access to EDs. A third category of study might look at challenges to caring for vulnerable populations in traditional primary care settings. These studies would likely find that providers, as well as patients, can be victims of a complex medical care delivery system with limited resources, leading to a search for systems-level solutions to a complex series of problems.

Back to Article Outline

How does the study address the implicit assumptions in studying inappropriate ED use? 

Choosing to study ED use for problems that could be treated elsewhere risks implying that such ED use is unequivocally bad. Many researchers introduce their studies with statements that such ED use is costly, that it is inferior to care in traditional primary care settings because of the continuity of care offered in these settings, or that inappropriate ED use leads to ED overcrowding. However, in the United States, the marginal cost of ED care for minor problems is probably similar to the cost of care elsewhere,34 and the total cost for ED care in this country is only 3% of the US health care budget.35

Although there is evidence that patients with better continuity of care have lower ED use rates,36 to our knowledge, there is no evidence that patients who use the ED for minor problems have worse outcomes than patients who obtain care for the same problems in traditional primary care settings. Indeed, if patients use the ED for lack of a stable relationship with a primary care provider elsewhere, then referring them back to a primary care site that lacks continuity of care can hardly improve their outcomes.

ED overcrowding has multiple causes, including limited inpatient bed availability and problems with other resources, which may not respond to reducing the number of ED patients who can be seen and discharged home.37, 38, 39, 40, 41 Researchers who choose to study inappropriate ED use to reduce it must explain how reducing ED use will serve patients or society.

Back to Article Outline

Applying the standards for appropriateness research 

How do the articles1, 2 in this issue of Annals measure up to these standards? The international perspective on inappropriate ED visits is a valuable addition to the literature because it illustrates the extent to which other countries wrestle with the same issue that has consumed so much attention in the US literature. Furthermore, these articles illustrate that the issue does not immediately disappear with the advent of universal health insurance.

But how valid are the approaches taken in these 2 articles? Regarding the definition of appropriateness, the Spanish article1 describes, in 2 detailed appendixes, the Hospital Urgencies Appropriateness Protocol and its validation. The reliability of the instrument was excellent. However, its validity compared with expert opinion was poor (κ = 0.39) because experts classified more visits as inappropriate. Of note, the instrument might have less validity in the United States; for example, it classifies an ankle sprain that was radiographed as appropriate but a facial laceration as inappropriate.

The Portuguese article2 uses a definition of appropriateness modified from our own work.3 However, we combined the explicit criteria on which Figure 1 of this article is based with implicit criteria (ie, expert review), and the authors did not use expert review. We have subsequently shown the poor agreement between the explicit criteria used alone and other measures of appropriateness.6

The authors1 of the Spanish article acknowledge the societal context of appropriateness in discussing their conceptual framework (in their Appendix 2). The Portuguese article2 lacks any discussion of the societal context of appropriateness. Unfortunately, both articles compare inappropriate with appropriate ED visits, rather than comparing inappropriate ED visits with similar visits to primary care providers. We will not comment on the selection of variables or the implicit assumptions because we lack the requisite knowledge of the Spanish and Portuguese medical care systems to understand what would best inform policy in these countries.

Although our lack of expertise in the medical care delivery systems of Portugal and Spain limits our ability to make definitive comments about these 2 articles,1, 2 we caution the reader about applying these findings in the United States. We urge US readers to critically review research on inappropriate ED use. This literature runs the risks of applying unvalidated classification schemes for appropriateness, ignoring lack of access to alternative sites of care, misinterpreting associations between patient characteristics and ED use, blaming the victims of poor access for their decisions to use the only remaining safety net, and falsely assuming that ED use represents inferior care at a higher cost. There are better ways to serve our patients than to blame them for seeking our help.

Back to Article Outline

References 

  1. Sempere-Selva T, Peiró S, Sendra-Pina P, et al.  Inappropriate use of an accident and emergency department: magnitude, associated factors, and reasons—an approach with explicit criteria. Ann Emerg Med. 2001;37:568–579
  2. Pereira S, Oliveira e Silva A, Quintas M, et al.  Appropriateness of emergency department visits in a Portuguese university hospital. Ann Emerg Med. 2001;37:580–586
  3. Lowe RA, Bindman AB, Ulrich SK, et al.  Refusing care to emergency department patients: evaluation of published triage guidelines. Ann Emerg Med. 1994;23:286–293
  4. Lowe RA, Chhaya S, Nasci K, et al.  To whom do managed care providers deny ED care?. [abstract] Acad Emerg Med. 1995;2:371
  5. Abbuhl SB, Lowe RA. The inappropriateness of “appropriateness”. Acad Emerg Med. 1996;3:189–191
  6. Lowe RA, Bindman AB. Judging who needs emergency department care: a prerequisite for policy-making. Am J Emerg Med. 1997;15:133–136
  7. Young GP, Lowe RA. Adverse outcomes of managed care gatekeeping. Acad Emerg Med. 1997;4:1129–1136
  8. McCaig LF. National Hospital Ambulatory Medical Care Survey: 1992 Emergency Department Summary.. Advance data from vital and health statistics; No. 245 Hyattsville, MD: : National Center for Health Statistics; 1994;
  9. General Accounting Office US. Emergency Departments: Unevenly Affected by Growth and Change in Patient Use. Washington, DC: : US Government Printing Office; 1993; GAO/HRD-93-4
  10. Dickhudt JS, Gjerdingen DK, Asp DS. Emergency room use and abuse: how it varies with payment mechanism. Minn Med. 1987;70:571–574
  11. Gifford MJ, Franaszek JB, Gibson G. Emergency physicians’ and patients’ assessments: urgency of need for medical care. Ann Emerg Med. 1980;9:502–507
  12. Guterman JJ, Franaszek JB, Murdy D, et al.  The 1980 patient urgency study: further analysis of the data. Ann Emerg Med. 1985;14:1191–1198
  13. Buesching DP, Jablonowski A, Vesta E, et al.  Inappropriate emergency department visits. Ann Emerg Med. 1985;14:672–676
  14. Haddy RI, Schmaler ME, Epting RJ. Nonemergency emergency room use in patients with and without primary care physicians. J Fam Pract. 1987;24:389–392
  15. Jacobs AR, Gabett JW, Wersinger R. Emergency department utilization in an urban community: implications for community ambulatory care. JAMA. 1971;216:307–312
  16. Rydman RJ, Mullner R, Vaux K, et al.  Using the emergency department medical record in financial and epidemiologic research. J Health Information Manage. 1992;1:46–58
  17. Nelson DAF, Nelson MA, Shank C, et al.  Emergency room mis-use by medical assistance patients in a family practice residency. J Fam Pract. 1979;8:341–345
  18. Berns SD, Linakis JG, Lewander WJ, et al.  Appropriate use of a pediatric emergency department: is the pediatrician called before the visit?. Pediatr Emerg Care. 1994;10:13–17
  19. DeAngelis C, Fosarelli P, Duggan AK. Use of the emergency department by children enrolled in a primary care clinic. Pediatr Emerg Care. 1985;1:61–65
  20. Chyba MM. National Center for Health Statistics: utilization of hospital emergency and outpatient departments, United States, January-June, 1980. In: National Medical Care Utilization and Expenditure Survey, Preliminary Data Report No. 2. Washington, DC: : US Government Printing Office; 1983;
  21. Yarboro TL. Emergency room use by patients from the family practice of a black physician. J Natl Med Assoc. 1990;82:93–97
  22. Australasian College for Emergency Medicine . National triage scale. Emerg Med (Australia). 1994;6:145–146
  23. Beveridge R. CAEP issues. The Canadian Triage and Acuity Scale: a new and critical element in health care reform. Canadian Association of Emergency Physicians. J Emerg Med. 1998;16:507–511
  24. Wilson FP, Wilson LO, Butler AB, et al.  Algorithm-directed triage of pediatric patients: a prospective study. JAMA. 1980;243:1528–1531
  25. Wilson LO, Wilson FP, Canales L. Algorithm-directed triage in a pediatric acute care facility: a retrospective study. Ann Emerg Med. 1981;10:427–431
  26. Wilson LO, Wilson FP, Wheeler M. Computerized triage of pediatric patients: automated triage algorithms. Ann Emerg Med. 1981;10:636–640
  27. Derlet RW, Nishio DA. Refusing care to patients who present to an emergency department. Ann Emerg Med. 1990;19:262–267
  28. Sackett DL, Straus SE, Richardson WS, et al.  Evidence-Based Medicine: How to Practice and Teach EBM. 2nd ed. Edinburgh, Scotland: : Churchill Livingstone; 2000;
  29. Foldes SS, Fischer LR, Kaminsky K. What is an emergency? The judgments of two physicians. Ann Emerg Med. 1994;23:833–840
  30. Brillman JC, Doezema D, Tandberg D, et al.  Triage: limitations in predicting need for emergent care and hospital admission. Ann Emerg Med. 1996;27:506–508
  31. O’Brien GM, Shapiro MJ, Woolard RW, et al.  “Inappropriate” emergency department use: a comparison of three methodologies for identification. Acad Emerg Med. 1996;3:252–257
  32. The Medicaid Access Study Group . Access of Medicaid recipients to outpatient care. N Engl J Med. 1994;330:1426–1430
  33. Cunningham PJ, Clancy CM, Cohen JW, et al.  The use of hospital emergency departments for nonurgent health problems: a national perspective. Med Care Res Rev. 1995;52:453–474
  34. Williams RM. The costs of visits to emergency departments. N Engl J Med. 1996;334:642–646
  35. Centers for Disease Control and Prevention . Medical-care spending–United States. MMWR Morb Mortal Wkly Rep. 1994;43:581–586
  36. Gill JM, Mainous AG, Nsereko M. The effect of continuity of care on emergency department use. Arch Fam Med. 2000;9:333–338
  37. Andrulis DP, Kellermann A, Hintz EA, et al.  Emergency departments and crowding in United States teaching hospitals. Ann Emerg Med. 1991;20:980–986
  38. American College of Emergency Physicians . Measures to deal with emergency department overcrowding. [position statement] Ann Emerg Med. 1990;19:944–945
  39. Fadale J. Overcrowding–comfort, consideration, convenience. J Emerg Nurs. 1990;16:132–133
  40. Lenehan CP. ED overcrowding and blaming the victim. J Emerg Nurs. 1989;15:211–213
  41. Lynn SG, Kellermann AL. Critical decision-making: managing the emergency department in an overcrowded hospital. Ann Emerg Med. 1991;20:287–292

☆ Reprints not available from the authors. Address for correspondence: Robert A. Lowe, MD, MPH, University of Pennsylvania Medical Center, Center for Clinical Epidemiology and Biostatistics, 914 Blockley Hall, 423 Guardian Drive, Philadelphia, PA,19104-6021; 215-898-0845, fax 215-573-2265; E-mail rlowe@cceb.med.upenn.edu. Address for correspondence after July 1, 2001: Robert A. Lowe, MD, MPH, Oregon Health Sciences University, Department of Emergency Medicine,UHN-52, 3181 SW Sam Jackson Park Road, Portland, OR 97201-3098;,503-494-7500, fax 503-494-4997; E-mail lowero@ohsu.edu.

PII: S0196-0644(01)62277-7

doi:10.1067/mem.2001.115216

Refers to article:

  • Inappropriate use of an accident and emergency department: Magnitude, associated factors, and reasons—An approach with explicit criteria

    Teresa Sempere-Selva, Salvador Peiró, Pilar Sendra-Pina, Consuelo Martínez-Espín, Inmaculada López-Aguilera
    Annals of Emergency Medicine June 2001 (Vol. 37, Issue 6, Pages 568-579)

  • Appropriateness of emergency department visits in a Portuguese University hospital

    Sofia Pereira, António Oliveira e Silva, Manuel Quintas, Jorge Almeida, Cristina Marujo, Manuel Pizarro, Vítor Angélico, Luisa Fonseca, Eunice Loureiro, Sofia Barroso, Augusta Machado, Marta Soares, António Braga da Costa, A.Falcão de Freitas
    Annals of Emergency Medicine June 2001 (Vol. 37, Issue 6, Pages 580-586)

Annals of Emergency Medicine
Volume 37, Issue 6 , Pages 629-632, June 2001

Article Tools