Suggestions for Improving the Reporting of Clinical Research: The Role of Narrative

The Absence of Narrative 

Expert panels, in recognition of problems in the reporting of randomized trials, meta-analyses of randomized trials, meta-analyses of observational studies, and evaluations of diagnostic tests, have created CONSORT, QUOROM, MOOSE, and STAR-D—guidelines that attempt to improve quality through standardization.12, 13, 14, 15 Although these have improved the quality of reports,16 when used only as checklists the guidelines may improve the completeness of an article as judged by these standards while doing little to ensure an honest and adequately comprehensive description of the research. This is because the guidelines do not address the narrative qualities of an article. Every scientific investigation is a story, and it is the details of the story that give it its richness.

If word limits were once responsible for the dearth of narrative in reports of original research, that excuse no longer holds because there is ample space on the Internet for online-only appendices.17, 18 If fear of subjectivity is responsible for the paucity of narrative, then the recognition during the past century that objectivity is an unobtainable concept should mitigate this fear. Indeed, it is difficult to find reasonable justification for not including descriptions of the investigators' thought processes in the report of an investigation. There are 4 topics for which added narrative would be particularly welcome. First, articles can describe the decisions that transpire between the conception of the crude idea and the finalization of the study design. Second, authors can provide a fuller account of how the study as executed differed from the study as planned. Third, authors can explain the assumptions invoked by the analyses used to reach their stated conclusions, and fourth, for studies that create models, authors can describe the rationale for the method of model development. I consider these in turn.

From Crude Idea to Study Design 

All clinical research and most bench research require compromise. An investigator begins with a general question he or she truly wants to answer (Figure 1) and envisions a study that will support a wide-reaching, definitive conclusion. Unfortunately, this ideal study can seldom be done. Ethical, financial, logistic, and technical constraints necessitate a trimming of the grand plans.

View Large Image Download to PowerPoint

Figure 1. From first ideas to study to conclusions: a schematic. This schematic documents the transformation of an idea into a study from which conclusions are drawn. Constraints prevent the design and execution of a study that would require only modest assumptions to logically support definitive conclusions (see text). Instead of reporting the modest conclusions supported by the actual study, investigators often offer more expansive claims. To do this, they make additional “bridging” assumptions (the right side of the Figure). Because the validity of these stronger conclusions is wholly dependent on the validity of the bridging assumptions, it is essential that investigators explicitly define and justify them.

Ethical constraints include the need to obtain informed consent (and the selection bias created when eligible subjects refuse), the need to ensure that members of the control group are not denied beneficial treatment, and the need to inform the subjects of what they may be subjected to (thereby changing the way they may react to the study instruments). Financial constraints limit, among other things, the size of the study, the number of measurements that can be made, the types of measurements that can be made, and the number and geographic dispersion of enrollment centers. Many logistic constraints are related to financial limitations, including the availability of staff to enroll, follow, and assess subjects at any time of day; the ability to ensure that subjects are in the right place at the right time to complete all aspects of the study protocol; and the ability to get all data points recorded and transcribed accurately. Other logistic constraints are independent of finances, such as the inability to adequately blind a study because there is no feasible sham control. Technical constraints include imperfections in measurements made by electronic equipment and survey instruments.

Designing a study is a gruesome process in which parts of the initial plan must be hacked off to preserve the whole. The investigator makes a host of decisions, balancing competing interests to make the most compelling and honest study the budget can attain. By describing these decisions and the rationale for making them, investigators help readers in 2 ways. First, they help readers understand the study's potential shortcomings, why they exist, and how they might affect interpretation of the results. Second, they give researchers practical insights that may help them plan their own research.

What Really Happened 

Anyone who has participated in clinical research knows that despite the best intentions of organized, intelligent investigators, few studies go off exactly as planned. The hordes of eligible patients available during the pilot phase miraculously disappear; participants forget to fill out forms or stumble over misunderstood items; eager house officers treat 30 study patients each while others treat none; the planned analytic strategy, based on the assumption of normality, must be altered to accommodate skewed distributions. Yet someone whose knowledge of clinical research was based solely on reading the literature would think these events were rare. This divergence of the literature and reality cannot serve science well. I recognize that it will be the rare investigator who is candid, confident, and alas, foolhardy enough to write “the study was stopped early so that it could be published and added to my c.v. before I went up for tenure,” but any shift toward more truthful accounts would be welcome.19 The benefits of such honesty are self-evident.

Conclusion, Assumptions, and Limitations 

Whether contained in a separate section (as in Annals of Emergency Medicine) or occupying a few paragraphs in the Discussion, it is now customary that reports include a consideration of the study's limitations.1 Authors commonly organize this material into 2 sections, one that considers threats to internal validity and one that considers threats to external validity.20 An alternative or complementary structure is to separately consider limitations arising from the study design, study execution, and analytic strategy. Yet another, and potentially better, alternative is to discuss these issues in terms of “assumptions” rather than “limitations” because conclusions can be reached only by combining data with assumptions.21, 22, 23

Regardless of which organizational paradigm is chosen, 2 issues should be kept in mind. First, the purpose of the Limitations section is to help readers understand the cautions that must be taken when interpreting study results. A narrative that lists a large number of potential limitations but fails to consider how they affect interpretation of results does not serve this purpose well. Instead, authors should discuss each important limitation and consider how it might affect the study's conclusions. This can often be accomplished through formal sensitivity analyses that are described in the Methods, reported in the Results, and discussed in the Limitations.1, 24

Second, authors must appreciate that, despite being set off in a separate section, limitations are not separate entities. They exist only as dyads such as “limitation: conclusion” and “limitation: research question.” Consider a perfectly designed and executed trial comparing drugs A and B in men. The limitation that the study did not include women is not a limitation if the research question was “which drug works better in men” or the conclusion was “drug A is better than drug B in men.” In other words, one can state limitations only in the context of a specific research question or conclusion. The parsing of limitations and conclusions into separate sections of research papers, although beneficial to the organization of the article, is artificial and can lead to misrepresentation and misinterpretation.

In particular, authors must take care to include important caveats (eg, “in men”) in the conclusion rather than only mentioning them in the Limitations. A report of the aforementioned study that concluded “drug A is better than drug B” would be dishonest, even if the Limitations section began, “Our study is limited by our failure to enroll women.” This technique—writing expansive conclusions and backtracking in the Limitations section—creates great potential for misrepresentation because some readers will not read the entire article.

The decision to include a caveat in the conclusion or discuss it in the Limitations requires judgment. Not every situation is straightforward. If the above study had 4% missing outcome data, it might be wholly appropriate to leave the conclusion unchanged (“drug A is better than drug B in men”) and discuss how the missing data could alter conclusions in the Limitations section. On the other hand, if there were 30% missing data, such a strategy could be misleading. Again, a narrative that explains the situation and examines its ramifications is the best strategy.

Describing Model Construction 

Many articles present mathematical models that include independent variables selected from a group of candidate variables. Such efforts require important decisions in 3 areas: model specification (eg, is the relationship among variables assumed to be linear, logistic, fractional polynomial), variable selection (which variables are to be included in the model), and variable specification (eg, should the variable “age” be entered as years, a binary variable [<40 or ≥40], or a categorical variable [<12, 12-49, 50-69, >69]). Decisions in each of these domains can have profound effects on the validity of the final fitted model, yet articles seldom describe how these decisions are made.

Specifying the model form (eg, 2 variables are linearly related over the range of interest) is a huge assumption, and authors should describe the basis for this decision. The widespread availability of sophisticated statistical software has increased the temptation to use statistical significance as a basis for variable selection, and investigators often use some form of mechanical statistical rule (stepwise regression, subset regression, univariate analysis to determine statistically significant candidate variables followed by multiple regression using those variables) to determine which variables are included in the model. There is a large literature that counsels against this and argues that theory should guide the process.25, 26 Regardless of one's feeling about this ongoing debate, it is difficult to argue how narrative that explains the rationale for these processes could be anything but helpful, especially if placed in an unobtrusive online-only technical appendix.

Other decisions about modeling warrant discussion. Excluding a factor or variable from a model is equivalent to making the strong statement that “this factor (variable) has no effect,” or “I am certain that the coefficient for this variable is zero”; hence the adage, “the most important thing about a model is what's left out.”27 Excluding cross-product (interaction) terms from a regression or analysis of variance (ANOVA) model implies that the relationship of one variable to the outcome is constant across all strata of the other variables.28 This is another strong (but often unrecognized) assumption that should be acknowledged and justified. Finally, the importance of checking on one's model for goodness of fit and other regression diagnostics cannot be overemphasized.29, 30

Where to put it all 

Until the 3- to 7-page printed research publication is replaced by electronic formats that allow readers to select the degree of detail they desire, authors will have to make difficult decisions about what material to include in the printed article.31 The narrative material suggested here will not all fit in a standard-length article, and most readers do not want to read anything longer. Authors need to balance comprehensiveness with readability. The material described in this article can be placed in several places, including:

I have argued for the importance of supplementary narrative in the presentation of clinical research. My argument could be extended to suggest that, like the BMJ and others, all journals abandon the standard 3- to 7-page journal article as the official version of an article in favor of a more comprehensive electronic format.17, 31, 32 Until this is done, authors need practical advice about how to include narrative descriptions that address Annals' Instructions for Authors request that authors “describe what you did and why you did it.”1 Three examples follow: a report of a randomized controlled trial, a report of an observational study, and an evaluation of a diagnostic test. Regardless of the study design, the principle is that articles (or their appendices) contain narrative that explains why the authors studied their question the way they did; why they made the choices they made; and how those choices, and the assumptions implicit in them, affect and limit interpretation of the results.

Example 2: Reporting an Observational Study 

Most observational studies will require additional narrative to consider the confounding introduced by the nonrandomized design. To adequately address the possibility of confounding, studies should contemplate a model of all measured and unmeasured factors that might directly or indirectly affect the outcome. Authors can explain what variables are used to represent each factor in their model. Without such a theoretical model, it is difficult to disentangle causal effects and to properly interpret the study. Authors may find graphic methods such as causal diagrams helpful in explaining the theoretical model for observational studies.27, 33 This material can be placed in an online-only appendix or in the article's Methods.

The following is an example of how a database study might be described in an article.

Somewhere in the Methods

The theory and detailed methods for the development of our model can be found in online-only Appendix 1 [url]…

Online-Only Appendix 1

We were interested in examining what factors might explain why certain patients have multiple visits to the emergency department (ED). We began with the Anderson and Aday model of health care utilization. [ref] This model identifies the environmental factors, predisposing characteristics, enabling resources, and health care needs that are associated with individuals' health care utilization. We customized this model to include the additional variables: chief complaint, frequency of receiving narcotics during each ED visit, and percentage of ED visits that resulted in hospitalization; factors that we postulated may be related to frequency of visits. Although we were interested in the “frequent flier” problem in general, we chose to study 3 tracer conditions, headache, back pain, and sickle cell painful crisis, because we thought these conditions were easily defined and therefore could be used to match patients by condition. We estimate that more than 75% of our frequent visitors offer one of these complaints as their predominant problem.

Figure 2 shows our conceptualization of the problem; variables in color and in bold are considered in our analysis. A discussion of all variables can be found in Figure 3. Our definition of cases and controls is designed to identify patients with similar demographics who present with similar complaints, but at different frequencies.

View Large Image Download to PowerPoint

Figure 2. Model of factors related to frequent ED visits. This graphic depicts factors thought to affect whether someone uses the ED with high frequency. We matched 3 control (1 to 5 visits/year) subjects to each case (>8 visits/year) on the green variables (see text for matching algorithm). We were particularly interested whether narcotic administration is associated with frequency of visits but included all bold variables in the model. Variables in regular typeface are potentially important but were not measured. An explanation of how variables were entered into our model is contained in Figure 3. SES, Socioeconomic status; PMD, private medical doctor.

Figure 3

Variables included in our models. For a published example of a table like this, see Sun B, Brinkley M, Morrissey J, et al. A patient education intervention does not improve satisfaction with emergency care. Ann Emerg Med. 2004;44:378-383.34

Outcome variables

Frequency of ED use is kept as 2 variables, 1 binary to indicate whether the subject is in the high-use (1) or low-use (0) group. The other variable contains the number of visits to the study ED in the past 12 months.

Matching variables

Age was categorized as 16 to 25, 26 to 49, and 50 to 70 years because we postulated that patients would be homogeneous within these strata but different among them.

Chief complaint was categorized as headache, sickle cell disease, or back pain.

Exposure variables

Number of narcotic doses per patient per visit is captured by 3 variables, the average number of doses per ED visit, the median number of doses per visit, and the maximum number of doses per visit.

Total narcotic dose per patient per visit is captured by the same 3 variable classes, with all drugs converted to morphine equivalents.

Length of stay is the median time in the ED during all visits for that patient.

Admitted is recorded as a binary variable for admission (1) or all other departures (0) (discharged, left without being seen, eloped)

(and so on for the remainder of the variables)

LIMITATIONS (abridged)

…Our findings suggest that having frequent visits was associated with receiving narcotics more often and at higher doses. The validity of our estimates of these associations depends on accurate classification of each subject's frequent visit status, which could be compromised by our failure to get data from all neighboring EDs. Readers should remember that an observational study design cannot establish causation….