Measures of Emergency Department Crowding, Odds Ratios, and the Dangers of Making Continuous Data Categorical: Answers to January 2008 Journal Club Questions

Discussion Points 

1. The authors performed a retrospective cohort study of all patients with severe pain presenting to their emergency department (ED). Why do you think the authors chose this type of study design? Describe the strengths and weaknesses of this design and contrast these with those of a prospective cohort study. If you were replicating this study prospectively, how might you alter the study design with respect to: a) the assessment of severity of patient pain; b) receipt of pain medication in the ED; and c) the measurement of ED crowding?

2. In Table 1, the authors report the mean age and standard deviation of patients and the median total patient care hours with the interquartile range. What do the mean and median values of a dataset represent and why might authors choose to report the median value instead of the mean value? What types of graphics can be used to display the distribution of a continuous variable such as age? Why, traditionally, have summary statistics been preferred to graphs?

3. The authors report that only 49% of patients judged by staff to have severe pain received pain medication while in the ED. What factors seemed to be most predictive of not receiving pain medication while in the ED? Why might it be important to know the frequency of these factors in the study population? Based on your own clinical experience, does it seem plausible that nearly half of patients with severe pain received no pain medication in the ED? If not, what might be responsible for this finding? If you were the principal investigator and were concerned about the validity of this finding, how might you verify its accuracy? Define the terms “internal validity” and “external validity” and explain how these terms specifically apply to this study's conclusions.

4. Pines and Hollander comment in the Discussion that “severity is associated with higher odds of non-receipt of any pain medication, possibly because providers focus more on diagnosis than symptom control.” How might you design a study to test whether physicians prioritize diagnosing the patient's condition over patient pain control? The CEO of your hospital reads this article and charges you with designing system changes to correct this problem. Describe what you might do to increase the likelihood that a patient with severe pain would promptly receive analgesia.

5. In Table 2, the authors report the odds ratios for time to analgesia based on measures of ED crowding. The odds ratio for waiting room number was 1.03 (95% confidence interval 1.02-1.03) to predict “no analgesia given in the ED.” Why did the authors report odds ratios? What assumptions underlie these estimates of the odds ratio? In this study, the odds ratio of 1.03 is comparing the odds of no analgesia when N patients are in the waiting room with the odds of no analgesia when N+1 patients are there. How much do the odds of no analgesia increase when there are 5, 10, 15, or 20 additional patients in the waiting room? If the authors instead chose to use an increase of 5 additional waiting room patients as their metric how would this alter the odds ratio? What do these values mean to you as an emergency physician? Can you calculate (approximately) the probability of not receiving analgesia as a function of waiting room occupancy?

6. In your opinion, what are the most important conclusions from this paper? How might the limitations mentioned by the authors affect your decision whether to change your clinical practice with regard to administering analgesia to patients during times of ED crowding? What additional information or data analyses would you like the authors to provide in order for you to change your clinical practice?

Answer 1 

Q1.1 The authors performed a retrospective cohort study of all patients with severe pain presenting to their emergency department (ED). Why do you think the authors chose this type of study design?

The authors' institution has a computerized patient charting and order entry system (EMTRAC) that fosters the collection of standardized information on each patient and stores that information in a relational database. Retrospective cohort studies have the advantages of low cost and rapid execution but can only be undertaken when adequate data have been collected as part of routine operations. In this case, information about pain severity, time to analgesia, and ED census was available in the EMTRAC system, making a retrospective cohort study feasible.

Q1.2 Describe the strengths and weaknesses of this design and contrast these with those of a prospective cohort study.

As noted above, retrospective cohort studies are inexpensive and can produce results very quickly. Furthermore, when done properly, cohort studies have the advantage of including a wide spectrum of patients with different severities of disease and different comorbidities who are treated by typical practitioners. Cohort studies can better approximate reality than randomized trials that are typically performed on highly selected patient populations in unique or atypical settings.

In a retrospective cohort study, the cohort is declared after the exposure and outcome have occurred, so there is often no control over data collection. (The definitions of “prospective” and “retrospective” studies are by no means standardized. For a general discussion of this topic, see Rothman and Greenland1). For example, in 2008 a researcher could declare his or her cohort to be those who graduated from Ivy League colleges between 1970 and 1980 but could not state, “and I want to measure each graduate's smoking history every 5 years after graduation.” He or she could, however, conduct analyses using available data.

In a prospective cohort study, the cohort is assembled and data elements are defined before data collection, so there is the opportunity to define what data will be collected and how, ie, “Every 5 years, we will telephone every person who graduated from an Ivy League school in 2009 and ask about their smoking habits and whether they have lung cancer.”

Thus, prospective cohort studies offer the investigator much greater control of who is studied and how they are studied. The disadvantage is that such studies must go on for years if the disease or outcome is slow to develop and therefore require a huge, well-funded infrastructure capable of tracking subjects over space and time in our highly mobile society.

Q1.3.a If you were replicating this study prospectively, how might you alter the study design with respect to the assessment of severity of patient pain?

It would seem that the question that these investigators would like to answer is, Did those who desired pain medication receive that medication in a timely manner? If this study were being done prospectively, investigators could arrange to have the triage nurse directly ask the patient, Do you desire medication for pain? Furthermore, patients could periodically be reassessed to determine whether their pain had been successfully treated.

From the article's Table 1, we learn that the triage nurse assigned a level 4 triage (low acuity) to 17% of patients who reported that their pain was severe. One interpretation of this finding is that the triage nurse's assessment of the patient's pain was discordant with the patient's self-assessment. In a prospective study, nurses could be asked to indicate whether they believe that the patient would benefit from treatment for pain.

Q1.3.b If you were replicating this study prospectively, how might you alter the study design with respect to receipt of pain medication in the ED?

There are 2 issues about the pain medication data. First, as the authors acknowledge, there may be discrepancies between the actual administration time of the pain medication and the time recorded by the nurse. In a prospective study, methods could be developed to ensure these times were accurate. Second, there may be some confusion about what constitutes treatment for pain. For example, it appears that a patient with angina whose 10 of 10 chest pain was completely alleviated with sublingual nitroglycerin would be counted in the “not treated” group in this study. In a prospective study, better procedures for determining whether a patient's pain was adequately managed could be developed that would eliminate the ambiguities created when the treatment of pain is judged solely by the administration of a limited list of medications.

Q1.3.c If you were replicating this study prospectively, how might you alter the study design with respect to measurement of ED crowding?

A large number of measures have been developed to assess crowding, and it is likely that they measure different aspects of this construct.2, 3, 4, 5, 6, 7 In addition to hard measures of occupancy, it might be useful to get gestalt assessments of crowding from the charge nurse, the triage nurse, or the attending physician. Although it is hoped that these subjective judgments would correlate with “hard” measures, it is certainly possible that there would be differences among measures and the subjective measures might better capture the likelihood of delay in the provision of analgesia.

Answer 2 

Q2.1 In Table 1, the authors report the mean age and SD of patients and the median total patient care hours with the interquartile range. What do the mean and median values of a data set represent, and why might authors choose to report the median value instead of the mean value? What types of graphics can be used to display the distribution of a continuous variable such as age?

The mean, median, and mode are measures of central tendency; they provide a sense of where the middle of a distribution is. The arithmetic mean (or “average”) is calculated by summing all values and dividing by the number of values. The median is the middle value when the data are in numeric order. If there is an even number of entries, the median is the average of the middle 2 entries. The mode is the value for which there are the most entries.

For example, a study compared the days of pain relief provided by 3 arthritis medications, drugs A, B, and C. For simplicity, we consider a study with 5 persons per group. Recognize that with so few subjects, it would be inappropriate to pass judgment about many of the topics that follow (skewness, normality, etc) but we do so for illustrative purposes.

View Large Image Download to PowerPoint

The drug A times are bimodal, with a peak at 1 to 2 and another at 8 to 10, with nothing in between. The middle value, or median (the third when the 5 values are placed in numerical order), is 8. The mode is meaningless because there are 5 unique values. The mean is 6 (30/5), but no patients have this value. In this case, the mean—or any measure of the central tendency of this data—would be misleading. When data are not unimodal anything short of a graph of the data tends to mislead.

In drug B, the mean, median and mode are all the same. Note that the mean is the same as for drug A, yet the distributions are very different. Though these data are symmetrical about 6, we have too little information to decide whether this distribution is “normal.”

Drug C has the same mean as the others but a lower median (5 versus 6 and 8). This distribution is anchored around 5 but has a single high value (11), which pulls the mean above the median. Such distributions are “skewed.” The bulk of these observations are on the left (at lower values) and the distribution is “right tailed” or “right skewed.” With skewed distributions, the mean can be a misleading estimate of central tendency and median is often preferred. We again emphasize that a histogram of the data is better than either summary statistic. It shows exactly where the data fall. All measures of central tendency should be accompanied by an estimate of variance. That topic will be considered in a future journal club.

Pines and Hollander were kind enough to provide us with their data. Here is a tabular description of the variable “age,” which represents the age, in years, of each study subject.

View Large Image Download to PowerPoint

Here is a histogram of the data. A histogram is an excellent way of visualizing the distribution of continuous data.

View Large Image Download to PowerPoint

From the above table and graph (produced in Stata 10; StataCorp, College Station, TX) we learn that the mean age is about 39 years (solid line), a few years higher than the median (50th percentile) age of 36.7 years (dotted line). This suggests that the distribution is right tailed, and the graph confirms this. Most distributions involving time are right tailed because one cannot get to the left of zero-time, yet longer times are always possible.

We also learn that the 4 oldest patients in the data set are more than 120 years of age. We explore this further by asking STATA to list all patients older than 100 years:

View Large Image Download to PowerPoint

In a data set of more than 13,000 subjects, a few patients older than 100 years is plausible, but the largest 4 observations were probably entered incorrectly into the electronic medical record, and these errors could have been checked when the data set was created.

Similarly, a perusal of those younger than 10 years reveals that there are 2 very young patients in the data set and that one of them received 4 mg of morphine. No doubt the age for this subject was entered incorrectly.

View Large Image Download to PowerPoint

We recalculated the distribution of age dropping the 4 values greater than 120 years and the 2 values less than 14 years and produced from which we glean that the spurious values do not affect the data summary consequentially. We also note that none of the other variables in this data set had similar problems. Nevertheless, this example demonstrates that even experienced investigators can fail to detect spurious elements in their data and highlights the importance of looking at the distribution of every variable in both tabular and graphic format to ensure that the distribution makes sense.

View Large Image Download to PowerPoint

We now look at total patient care hours.

View Large Image Download to PowerPoint

We learn from the table and figure that the patient care hours distribution is skewed to the right, with the mean (127.6, dashed line) being higher than the median. The dotted lines and corresponding x axis labels signify 10th, 25th, 50th, 75th, and 90th percentiles of this distribution.

Q2.2 Why, traditionally, have summary statistics been preferred to graphs?

Before electronic publishing, it was very expensive to print figures because they had to be hand drawn and then etched into copper plates in a labor-intensive process. With the financial disincentive to publish figures gone, the remaining impediments are tradition and the absence of software that plops little histograms down in the middle of a table. There is no reason the first data table in an article should not look like:

View Large Image Download to PowerPoint

Figure 1. *Patient demographic and ED crowding measures (N = 13, 758). *This modified version of Table 1 in the Pines and Hollander paper demonstrates how small histograms can easily be inserted into a table.

Answer 3 

Q3.1 The authors report that only 49% of patients judged by staff to have severe pain received pain medication while in the ED. What factors seemed to be most predictive of not receiving pain medication while in the ED? Why might it be important to know the frequency of these factors in the study population?

According to Table 2 in the article, those older than 65 years and female patients were most likely to leave the ED without receiving pain medication, with odds ratios (ORs) of 1.65 and 1.30, respectively. Let us first consider what an OR of 1.65 means. It means that the odds of not receiving pain medication if you are older than 65 years is 1.65 times the odds of not receiving pain medication if you are younger than 65 years. Although the 1.65 value is an adjusted OR derived from a logistic regression model that accounted for other variables, let us calculate the unadjusted OR for the effect of age. The relevant table is:

View Large Image Download to PowerPoint

Our unadjusted OR (1.62) is quite similar to the adjusted OR reported in the article. A few features are noteworthy. First, only 962 persons are older than 65 years, a fact that is not directly available in the article. It is important to know the distribution of an explanatory variable in a data set because the frequency of the finding affects its importance. For example, if persons with 1 blue and 1 green eye never ever got analgesia the OR for this risk factor would be very, very high (infinite, in fact), but the factor would be unimportant because so few patients have this finding. Patients older than 65 years represent only 7% (962/13,758) of the patients in the Pines and Hollander study.

We should also consider whether taking a continuous variable such as age and making it into a categorical variable is wise. For illustrative purposes, we tabulate the percentage of subjects who received analgesia by age groups: younger than 20, 21 to 30, 31 to 40, 41 to 50, 51 to 60, 61 to 70, and older than 70 years. We find that there is actually a U-shaped distribution at which those at the extremes of age are more likely to leave untreated. This relationship is lost when the continuous variable age is dichotomized but preserved when age is partitioned into a larger number of categories.8 The new table makes us wonder if the “nitroglycerin problem” could explain the seeming failure to give pain medication to older persons and the real hypoanalgesia problem is in the young. Our main point, however, is that one needs to think carefully about what any particular OR means, paying special attention to the frequency of the risk factor in the population and the way that the risk factor was formulated.

View Large Image Download to PowerPoint

Q 3.2 According to your own clinical experience, does it seem plausible that nearly half of patients with severe pain received no pain medication in the ED? If not, what might be responsible for this finding? If you were the principal investigator and were concerned about the validity of this finding, how might you verify its accuracy? Define the terms “internal validity” and “external validity” and explain how these terms specifically apply to this study's conclusions.

We suspect that the clinicians at this study site were not pleased to find that 51% of patients who presented to their ED with 9 of 10 or 10 of 10 pain received no analgesia during their ED stay. Although this could mean that this ED is not doing a very good job treating pain, it is also possible that the 51% figure is inaccurate. Factors that could produce an inaccurate estimate include patients' refusal of medication, spontaneous reduction in pain score from triage to arrival in examination room, misinterpretation of patient pain score because of language barrier or erroneous computer entry, inaccurate documentation of the administration of analgesia, and physicians' overt decision not to provide analgesia despite patient's self-assessment. Investigators and administrators at this site might wish to examine to what extent these factors may have distorted the estimate.

The phenomena discussed in the previous paragraph all relate to the internal validity of the Pine and Hollander study. “Internal validity” and its partner “external validity” are 2 terms one should be familiar with when evaluating an article's results. Internal validity asks, If the study were repeated under the same circumstances using techniques that eliminated systematic bias and measurement error, would the same result be achieved? In other words, are there aspects of the study that render the result a biased estimate of the true result for that setting?

External validity asks whether the study's results apply to the implied population of the study question. For example, a study of the prevalence of chlamydia in Greenland might have perfect internal validity but could have poor external validity if the study question were, What is the prevalence of chlamydia in Central American men? There are a number of external validity issues with the Pines and Hollander study. One could easily imagine that results would be different had the study been conducted in a rural community ED or a private nonacademic medical center. Had this study been conducted in a large number of EDs with different geographic characteristics, we would be more confident that study findings were representative of US EDs. Such a study would have greater external validity than one conducted at a single site.

Answer 4 

Q 4.1 Pines and Hollander comment in the Discussion that “severity is associated with higher odds of non-receipt of any pain medication, possibly because providers focus more on diagnosis than symptom control.” How might you design a study to test whether physicians prioritize dianosing the patient’s condition over patient pain control?

The first step is to stratify patients by chief complaint, severity, or both. One would then examine the risk of hypoanalgesia in each stratum to try to understand whether patients with higher severity were truly not getting analgesia or whether this was a confounded observation (see discussion of chest pain treated with nitroglycerin in Question 1). If the severity effect could not be explained, it would be appropriate to conduct a study to understand why patients with severe illness were not receiving prompt treatment of their pain. Such a study should focus on a few common, high-severity complaints for which the reasons for not treating pain can be anticipated and measured. For example, in trauma patients the presence of hypotension might delay pain management, whereas some physicians may erroneously believe that providing analgesia to patients with abdominal pain might mask findings and delay diagnosis despite multiple studies that show that this is not the case.

Although it is possible that some electronic medical records are rich enough to permit such a study, it is more likely that a prospective study would be better in terms of feasibility and data quality. In a prospective study, the physicians who failed to treat pain could be asked why he or she did so. The danger of such a study design is contamination: By asking physicians why they did not do something, one would prompt them to do it either on the index patient or on subsequent patients.

Q 4.2 The CEO of your hospital reads this article and charges you with designing system changes to correct this problem. Describe what you might do to increase the likelihood that a patient with severe pain would promptly receive analgesia.

According to the results of a chief-complaint-based study, one could design system practice changes to improve the administration of analgesia in the ED. Such interventions might include standing orders for triage nurses to administer oral pain medications in triage to patients with severe pain, computerized alerts to remind physicians about analgesia when patients’ pain scores exceed a predefined level, or an in-service for emergency physicians and nurses about administering appropriate analgesia. Specific attention might need to be focused on patients with demographic characteristics or chief complaints that are associated with increased risk of delayed delivery of analgesia.

Answer 5 

Q 5.1 In Table 2, the authors report the ORs for time to analgesia based on measures of ED crowding. The OR for waiting room number was 1.03 (95% confidence interval 1.02 to 1.03) to predict “no analgesia given in the ED.” Why did the authors report ORs? What assumptions underlie these estimates of the OR?

First, we will define what each of these terms means.9, 10, 11 Except for devotees of Vegas or the ponies, physicians are generally more comfortable with probabilities (there is a 0.4 or 40% chance that it will rain today) than odds (the odds of rain are 4:6 (or 2:3 or 0.66)). These are all equivalent statements. Probability is calculated by dividing the chances an event will happen by the total number of chances. Odds compare the chances an event will happen to the chances an event will not happen. A look at the data table for age greater than 65 years and risk of not receiving analgesia (see answer to question 2) makes this concrete.

If one is older than 65 years, then the probability of not receiving analgesia is 597/962, or 62%. The odds are 597/365, or 1.6. Similar calculations for the younger group reveal a probability of 50% and odds of 1.0. We often refer to the probability as a “risk” because we are referring to the probability of a bad event. The probability ratio, or risk ratio (RR), is simply the ratio of the probabilities (risks) for each group. In this case, it is 0.62/0.5. or 1.24. Similarly, the OR is the ratio of the odds, 1.6/1.0, or 1.6. To change the reference group (ie, what the odds are of getting analgesia), one takes the reciprocal of the RR, 0.5/0.62=0.8, or OR, 1.0/1.6=0.625. Another metric that could have been used to report these data is the absolute risk difference, which is calculated by subtracting the risk for one group from the risk for the other, in this case, 0.62–0.5=0.12. In other words, those older than 65 years are 12% more likely to leave the ED without analgesia than younger patients. This metric has the advantage that its reciprocal is the number needed to treat to benefit or harm, a topic that will be discussed in a future journal club.

So why did Pines and Hollander choose to report their results as odd ratios? Why did they report that the OR for not receiving analgesia is 1.03 for each additional waiting room patient instead of reporting the RR or risk difference? In a case-control study, one cannot discuss risk, the RR, or the absolute risk difference, because the ratio of diseased to nondiseased has been set by the investigator. In this cohort study, however, this limitation does not apply and the authors could have chosen any of these metrics to report their findings. Most likely, they chose the OR and the adjusted OR because they used logistic regression as their analytic model and this type of model naturally produces ORs. We will discuss why in a future journal club.12

For now, what is important is to understand what assumptions are invoked when logistic regression and ORs are used in this way. The main one is that this model assumes that the additional risk incurred for each additional waiting room patient is constant regardless of whether the waiting room census is going from 1 to 2 or 40 to 41. Each additional patient increases the odds of no analgesia by 1.03-fold. Note also that the model assumes that the effect is multiplicative (1.03 times the previous odds), not additive (we add a constant amount to the odds for each additional patient). These are strong assumptions that may or may not be reasonable, depending on the circumstance. One could certainly make a strong theoretical argument that adding the first few patients to the waiting room might have little effect, as might adding an additional patient when the department is already overloaded. The effect is likely strongest when there are a medium number of patients in the waiting room. Just as we did above for the effect of age on oligoanalgesia, the best way to investigate these concerns is to stratify the data on how many patients are in the waiting room and then examine the odds of oligoanalgesia in each stratum. One of the difficulties for readers of articles that use models such as logistic regression is that the article seldom provides sufficient information to decide whether the model is reasonable. We must trust that the authors did things properly without any means of “looking under the hood” to see whether things were put together acceptably.

Q5.2 In this study, the OR of 1.03 is comparing the odds of no analgesia when N patients are in the waiting room with the odds of no analgesia when N+1 patients are there. How much do the odds of no anesthesia increase when there are 5, 10, 15, or 20 additional patients in the waiting room? If the authors instead chose to use an increase of 5 additional waiting room patients as their metric, how would this alter the OR?

As discussed above, the OR is a multiplicative measure. Thus, if the odds of not receiving analgesia are a when n patients are in the waiting room, the odds when there are n+1 patients is 1.03a, for n+2 patients is 1.03×(1.03 a), and the odds when there are k additional patients is 1.03k a. Therefore, the odds are 1.16-fold higher for 5 additional patients, 1.34-fold higher for 10, 1.56 for 15, and 1.88 for 20. When the OR for a contrast is close to 1 (a small increment in odds), it is sometimes helpful to change what is being contrasted in the ratio. In other words, rather than reporting the OR for no analgesia for each additional patient in the waiting room, the investigators could have reported the OR for every 5 additional patients. This number, 1.16, is a bit easier to understand and may be a bit more clinically relevant because adding 5 patients feels more substantial (more of a change to the ED's condition) than adding 1 patient.

Q 5.3 What do these values mean to you as an emergency physician? Can you calculate (approximately) the probability of not receiving anesthesia as a function of waiting room occupancy?

We have discussed how potential problems with external validity should make clinicians cautious about directly applying the results of this study to their ED. Those issues aside, what does it mean that the addition of 5 patients to the waiting room increases the odds of a patient not receiving analgesia by 1.16-fold? We can say that the odds are 16% higher than they were, but what were they? We know that the overall odds for not receiving analgesia were about 1.0 (49% did, 51% did not). We also know that the median number of patients in the waiting room was 8. If we assume that the average odds of no analgesia occurred when the waiting room had its typical number of patients, then we can argue that when 13 patients are in the waiting room, the odds are the OR times the present odds (1.16×1.0=1.16) and that these odds correspond to a probability of 1.16/(1.16+1)=1.16/2.16=0.537. In other words, one could roughly expect that with 13 patients in the waiting room, there would be a 54% chance of not receiving analgesia. We checked the percentage of untreated subjects who were seen when the waiting room had 8 or 13 patients and found:

View Large Image Download to PowerPoint

View Large Image Download to PowerPoint

We observe a steady increase in untreated patients as the waiting room number increases. At waiting room numbers above 30, the N is quite small so that the percentage is based on very small denominators (eg, fractions such as 1/1, 0/1, 1/2), making the graph quite staccato.

Answer 6 

6.1 In your opinion, what are the most important conclusions from this article? How might the limitations mentioned by the authors affect your decision whether to change your clinical practice with regard to administering analgesia to patients during times of ED crowding? What additional information or data analyses would you like the authors to provide in order for you to change your clinical practice?

From this article, we can certainly conclude that crowding was in part responsible for suboptimal pain management in this urban academic ED. For the many emergency physicians practicing in departments in which overfilled waiting rooms are routine and hallways are full of boarding patients, this conclusion will not be surprising. Nevertheless, studies that rigorously quantify how crowding diminishes quality are an important foundation for efforts to change local and national health policy.

Although the internal validity of this study may be imperfect (Question 3), it is unlikely that the study is so flawed that bias is responsible for the entire association between crowding and oligoanalgesia. We can be fairly certain that crowding is important in this ED. Nevertheless, it is important to consider that even when the waiting room was close to empty, 40% of patients never received analgesia (see the left side of the Question 5 Figure). This suggests that this particular ED needs to develop better methods for treating pain independent of issues specifically related to crowding.

But how well do the results of this study generalize to other EDs? Said another way, what is the external validity of this study? Until this study is replicated at other sites, this is a difficult question to answer. Some urban EDs may have protocols in place that ensure that patients receive treatment for their pain regardless of crowding status, so that crowding is not an important predictor of oligoanalgesia. At other EDs, crowding may have a greater effect on the delivery of pain medication than it did at the study hospital. This heterogeneity of effect no doubt exists in suburban and rural hospitals as well. We suspect that this study has poor external validity if the study question is, what is the typical relationship between crowding and pain medication delivery in US EDs, because this single study is unlikely to capture what is likely a very heterogeneous phenomenon. If the study question is, can the delivery of pain medication be used as an indicator of quality for documenting that crowding can be detrimental to quality care, then this study has good external validity.

In answering questions 1 to 5, we have suggested how the authors might have provided more information about the continuous variables in this study. In addition, readers might want to know what chief complaints were most or least associated with receipt of analgesia. Did the majority of patients with abdominal pain or orthopedic injuries receive analgesia, whereas patients with chest pain or chronic back pain did not? What factors led to the delay in administration of analgesia to patients once they were transported to a treatment room? Was the delay due to physicians not seeing the patients quickly, not ordering medications in a timely manner, or delay in the nurse giving the medication? It is likely that some of these questions could only be answered through a prospective study. The bottom line is that these investigators have demonstrated that in their ED, oligoanalgesia is associated with crowding and readers should contemplate whether this is a problem in their own EDs.