An evaluation of the graphical literacy of Annals of Emergency Medicine☆☆☆★

Article Outline

• Abstract
• Introduction
• Materials and methods
• Results
• Discussion
• Acknowledgements
• References
• Copyright

Abstract 

Study Objective: To describe the type, quantity, and quality of graphics used to present original research in Annals of Emergency Medicine. Methods: We performed a blinded, retrospective review of all graphics published in Annals of Emergency Medicine’s original research articles from January 1998 through June 1999. We assessed the types of graphics, the use of special features to display detail, the clarity of each graphic, discrepancies within the graphic or between the graphic and text, and the efficiency of data presentation. Results: Forty-six percent (68/147) of original research communications contained at least 1 graphic. Of the 128 graphics in these 68 articles, simple univariate displays predominated (53%). Only one third of graphics displayed by-subject data through the use of one-way plots, scatter plots, or other formats. Graphics generally defined all symbols and abbreviations (99%) and were self-explanatory (88%). Techniques for conveying the richness of a data set were seldom used (11% of all graphics). Forty percent (51/128) of the graphics contained internal contradictions (15%), muddled displays (19%), numeric distortion (5%), nonstandard graphing conventions (7%), and other lapses in design or execution. Inefficiencies of data presentation included internal redundancy (16%), extraneous decoration (10%), and redundancy of graphic data with other text/tables (15%). Conclusion: The majority of graphics in Annals of Emergency Medicine, although internally valid, failed to take full advantage of the graphic’s potential and often depicted summary data when portrayal of subject-specific data was possible. To help readers fully understand research findings, authors and editors should take care to ensure that graphics efficiently and effectively portray the optimal amount of information. [Cooper RJ, Schriger DL, Tashman DA. An evaluation of the graphical literacy of Annals of Emergency Medicine. Ann Emerg Med. January 2001;37:13-19.]

See related article, p. 75 .

Back to Article Outline

Introduction 

A well-known proverb states that one picture is worth a thousand words.¹ In scientific articles, an appropriate graphic can communicate research findings better than words or tables. A well-rendered graphic tells the story of the investigation, allowing the time-pressured reader to rapidly gain an understanding of the investigation’s results without reading the entire article, while providing the interested reader access to the details of the data. A graphic is often able to convey information that cannot easily be described in prose. The reader sees the data instead of reading a list of descriptive statistics summarizing them. In many instances, graphics can reveal results more precisely than conventional statistical computations.²

Understanding the importance of graphical data presentation is not sufficient to ensure a well-prepared manuscript, because not all graphics are created equal. The preparation of high-quality graphics requires an understanding of both what is to be communicated and human visual perception. In 1984, Cleveland³ proposed a 5-point agenda for research and development to improve graphical communication in science: (1) carrying out studies on how graphics are used, (2) developing methods for data presentation, (3) developing guidelines for graphic design, (4) studying human graphical perception, and (5) developing software for statistical graphics. This article addresses Cleveland’s first recommendation by describing the quantity, appropriateness, and quality of graphics in our specialty’s largest-circulation scientific journal, Annals of Emergency Medicine. Of note, in the 16 years since the publication of Cleveland’s proposal, to our knowledge only 2 investigators have addressed the graphical quality of medical manuscripts.³, ⁴ We describe the quality of Annals of Emergency Medicine’s articles not to pass judgment, but to characterize current graphical practice as a starting point for improvements in graphical literacy.

Back to Article Outline

Materials and methods 

For this retrospective review, we examined all articles presenting original research published between January 1998 and June 1999 in Annals of Emergency Medicine. We excluded case reports, correspondence, editorials, reviews, guidelines, policy statements, and methodology articles. We reviewed each article published during the study period to determine the presence of any data graphic and the number of such graphics per article. Articles presenting original research without graphics were included in the denominator of the percentage of research articles containing graphics statistic but were otherwise excluded from further analysis. Data tables and participant flow charts were not considered data graphics for the purposes of this review.

The purpose of a graphic is to display data “accurately and clearly.”⁵ The ideal graphic is free of errors, self-explanatory, presents details not easily conveyed with prose, and facilitates recall of the salient results. We chose objective criteria to reflect those characteristics of graphic design that we believed best measured these underlying concepts and coincided with the principles of well-designed graphics outlined by other authors.², ⁶, ⁷, ⁸, ⁹, ¹⁰, ¹¹, ¹², ¹³, ¹⁴, ¹⁵, ¹⁶

For each graphic, we addressed 3 constructs of interest. First, internal characteristics—is the graphic clear, self-explanatory, and internally consistent? Second, external characteristics—is the graphic consistent with the text, is it redundant with information in text or tables, and does it adequately portray the nuances of the data? Third, graphical efficiency—does the graphic convey the information in an efficient manner, maximizing information and minimizing extraneous noise?

We designed a 3-section data collection form, 1 section for each construct, with a coding manual containing specific definitions and operational criteria to score each construct’s elements. We refined the data collection instrument through 2 rounds of pilot-testing on Annals of Emergency Medicine articles published before 1998. Definitions and operational criteria are presented in Table 1. All criteria were judged using binary logic (yes/no or present/absent). Two reviewers (RJC and DAT) independently completed sections I and II of the review form on all the graphics. These results were analyzed for interrater agreement. Any disagreements were discussed among all authors and consensus achieved.

Table 1. Definitions and operational scoring criteria for evaluation of graphical quality.

A research assistant, not involved in evaluating the graphics, prepared materials for review. He photocopied each article that contained a graphic and then redacted all identifying information from the article. He also made “isolated graphics” (ie, individual photocopies of each graphic and its legend, separated from any adjoining page text, and shuffled them to create a random order).

To evaluate the first construct, we judged the “isolated graphics” without reference to the article’s text or figures. We noted the type of graphic (bar graph, histogram, scatter plot, and so on), and the presence of any special features, including the depiction of paired data, symbolic dimensionality, and small multiples. We inspected the graphics for internal contradictions (errors), completeness (explanation of symbols, abbreviations, and labels), numeric distortion, visual clarity, and adherence to standard graphing conventions (Table 1). We interpreted this last criterion leniently, not faulting graphics that deviated from standard norms if the legend or labels clearly indicated what conventions were used. One caveat to the “isolated” scoring of this section existed. For the question, “Were all abbreviations adequately defined?” the reviewers could mark the answer “yes,” “no”, or “pending article evaluation.” We later reviewed all of the “pending” responses and gave credit if the meaning of nonstandard abbreviations was obvious from the article’s title or abstract.

For the second and third constructs, we considered the graphic in the context of the entire article. To evaluate its external characteristics, we assessed whether there were any inconsistencies between the information presented in the graphic and that in any text/tables. We then determined whether the graphic represented the dimensionality measured in the study (eg, did the graphic fail to portray important stratifying variables), or whether paired data were appropriately depicted.

The third section of the review focused on the efficiency of data exhibition. We noted any extraneous decoration (“chartjunk”), and redundancy of data within the graphic itself. We appraised each graphic for redundancy with information in any text or table. There are sometimes advantages to viewing information in 2 formats, because a graphic can show the distribution and visual characteristics of the data, whereas a text or table can show the exact numeric values. Therefore, we only considered graphics redundant if they contained no new information beyond that presented elsewhere in the article. We calculated the data density index (DDI), intended to describe the average information content of a square centimeter of graphic²² (Table 1). We acknowledge that high data density is neither a necessary nor sufficient condition for graphical excellence. Nevertheless, the DDI does provide a useful proxy for the efficiency of information portrayal. In measuring this statistic, we did not give credit to points too cluttered to interpret.

Back to Article Outline

Results 

The 18 issues of Annals of Emergency Medicine contained 147 eligible articles presenting original research data; 79 (54%) had no graphics. Our article reports on the 128 graphics found in the 68 articles that contained at least 1 graphic. Thus, the denominator for all percentages was 128 unless otherwise stated. The number of graphics per article ranged from 1 to 5 (mean 1.9); the majority of articles had 1 (46%) or 2 (34%) graphics. Reviewers were concordant on 90% of elements in sections I and II of the data collection form. The most common discrepancies occurred when one reviewer failed to detect an internal contradiction in the graphic or when reviewers disagreed on whether the graphic was self-explanatory.

Univariate displays of descriptive statistics (eg, counts, percentages, or means) comprised 53% of Annals of Emergency Medicine’s graphics; bar graphs were most common (Table 2). We seldom encountered the depiction of paired data (7%), the use of symbolic dimensionality (2%), or the use of small multiples (2%) (Tables 1 and 3). Nearly all the graphics adequately defined their abbreviations, and most contained legends or other features to render the graphic self-explanatory. Internal contradictions, numeric distortion, nonstandard graphing conventions, or poor visual clarity compromised the interpretation of results in 40% (51/128) of the graphics (Table 3).

Table 2. Distribution of graphic types and frequency of special features among the 128 graphics in the study.*

Type of Graphic	%
Univariate displays	75
Categorical data (frequency counts or percentages)	26
Bar graphs†12, 13, 23	14
Point graphs†23	10
Pie charts12, 13, 23	2
Summary statistic of continuous data (eg, mean)	27
Bar graphs, without CIs	2
Bar graphs, with CIs	8
Point graphs of aggregate data without CIs	2
Point graphs of aggregate data with CIs	15
Distributions	22
Histograms10, 12, 13, 23	11
Box-and-whisker plots10, 23	3
Survival curves23	6
One-way plots9	2
Bivariate displays	23
Scatter plots12, 13, 23	11
Receiver operating characteristic curves9	7
Bland-Altman or Tukey plots7, 9, 24	5
Other (Venn, cumulative, Q-Q plots, etc)9, 10	2
Special graphic features
Depiction of paired data	7
Small multiples	2
Symbolic dimensionality	2
*The taxonomy of graphics is based on standard graphics texts and further described in the accompanying article.9, 10, 12, 13, 14, 15, 23 Additional information on specific graphic types can be found in the references after each entry. †Bar and point graphs are 2-axis graphics of summary points (either frequency counts or group means), measured along a continuous axis (typically the y-axis) versus a categorical variable or time (sometimes called a time plot) on the other axis. Point graphs are essentially bar charts, but the bar is eliminated and replaced with a point that designates the bar’s height. Point graphs are not classified as scatter plots, because the data points represent group-level responses, not subject-level responses. Despite the 2-dimensional portrayal with an x-axis and y-axis, both bar and point graph are considered univariate because the data points represent a single variable for a group, not the value of 2 different variables for a single subject.

CI, Confidence interval.

Table 3. Characteristics of the 128 graphics found in Annals of Emergency Medicine, January 1998 through June 1999.

IQR, Interquartile range (25th, 75th percentile).

In the analysis of external graphic characteristics, we found that discrepancies between the text and graphic (6%) were less common than contradictions within the graphic (15%). Overall, 33% (42/128) of graphics did not reveal the degree of detail appropriate for the investigations’ findings. Through an inappropriate choice of format, or failure to depict by-subject data, paired data, or confounders, these graphics failed to capture the detail, nuance, or dimensionality of the underlying data. Twenty-one percent of experiments generated paired data, yet only 33% (9/27) of these investigations’ graphics evinced the pairing.

We found poor efficiency of data presentation in many instances. Thirty-two percent (41/128) of Annals of Emergency Medicine’s graphics included either extraneous ink in the form of “chartjunk” (13/128), redundant information within the graphic and/or its legend (21/128), or redundancy with text (19/128). The DDI varied greatly depending on the graphic and the type of data analyzed in the study (Table 2). For all the graphic types combined, the median DDI was 0.94 cm^–2, less than 1 data point per square centimeter. Bivariate and univariate distribution graphics had the highest data densities.

Back to Article Outline

Discussion 

Our findings highlight the current state of graphical data representation in Annals of Emergency Medicine, and allow authors insight into areas for improvement. We do not contend that every article needs a graphic; in fact, we chose not to use any graphics to present the results of this paper, because there was nothing that could be portrayed more effectively in a graphic than in text and tables. Slightly less than half (46%) of articles during the 18-month review period present original research. The articles most often contained 1 or 2 graphics, less than reported in other medical journals.⁴ In the past, when journals were manually typeset, graphics were expensive to publish. Now that Annals of Emergency Medicine uses computerized page layout, the production costs are budgeted on a per-page basis, making no adjustment for anything but the use of color (personal communication with Cheryl A. Smart, Mosby, March 2000).

The majority of graphics evaluated were simple bar graphs, often denoting a descriptive statistic (mean or frequency). These findings are similar to previous reports of other medical science journals’ graphics.⁴ In general, these types of graphics are missed opportunities. The information presented in a bar graph of group means can more simply and succinctly be conveyed with prose: “the mean for group A was 24 and for group B was 31.” What these graphs omit—the actual data points or a summary of the data distribution—is not easily described by the written word. Although more detail is preferred, at a minimum, the addition of confidence intervals allows readers to assess the precision of findings. For studies designed with before and after measures, the depiction of pairing allows readers to understand what happened to each patient, not just the “average” patient. Symbolic dimensionality or small multiples allows readers to see both the main effect and how confounders or effect modifiers might alter results (Table 1). With symbolic dimensionality, we escape the constraints of the 2-dimensional paper plane and create graphics that show a general pattern plus specific detail, all within the same area. Not every graphic requires these extra levels of detail. The number of subjects, number of relevant variables, and distribution of the results should guide authors in determining the optimal amount of data necessary to depict the story of the investigation.¹⁶

Graphics in Annals of Emergency Medicine were generally well-labeled, with appropriate legends. Authors used clearly labeled axes, defined all symbols, and avoided numeric distortion, all of which facilitated the interpretation of the graphics. In a review of the anesthesiology literature, De Amici et al⁴ found similar rates for these statistics as we did, except for clear abbreviations in graphics, in which we found Annals of Emergency Medicine performed better. Similarly, the rate of redundant text/tables with graphic content noted to be 15% in our review was similar to the 2% to 14% range for the anesthesia literature.⁴ We found 15% of graphics had internal contradictions, and another 6% had discrepancies between the graphic and text. The combination of these 2 types of errors occurred in 1 of 5 Annals of Emergency Medicine’s graphics.

Notable in our review was a lack of efficient use of space and ink to convey data. Removal of extra lines and redundant numeric labels would improve the clarity of the graphic and let the data speak for themselves. Annals of Emergency Medicine’s DDI (0.94 cm^–2) is somewhat lower than that described for other medical publications. Tufte¹⁹ found a wide range of data densities in his evaluation of scientific literature (circa 1979-1980), with a median DDI of 1.86 cm^–2 in The New England Journal of Medicine and 1.09 cm^–2 in Journal of the American Medical Association. In contrast, during the same publication year, he found the journals Nature and Science had median data densities of 7.44 cm^–2 and 3.25 cm^–2, respectively, whereas Scientific American had only 0.78 cm^–2.¹⁹ The majority of graphics in Annals of Emergency Medicine were simple univariate displays of summary statistics (frequency counts or means), and these generally have lower data densities than scatter plots and other more granular formats.

Although the DDI statistic is an objective outcome measure, it only captures the amount of information per area and as such represents the first limitation of our study. The statistic does not take into account how much of the relevant, available data are portrayed. We chose to measure the DDI, because it has been reported in other journals and thus can be easily compared. However, we report this statistic with some hesitancy because it measures only one aspect of graphical quality.

Other limitations should be acknowledged. First, we did not attempt to evaluate articles that did not have graphics to determine whether graphics were warranted, nor did we penalize the articles we did review for failure to provide additional graphics that were desirable. It is possible that these omissions are a greater problem than the quality of the graphics that were presented. Second, we acknowledge that because authors do not control the final dimensions of the graphic as printed in a journal, the DDI is not solely based on the author’s submission, but reflects the journal’s typographic style.

In addition, we chose specific statistics as constructs of graphical quality and omitted other indexes. More detailed considerations of axis size, extent of necessary tick marks, text size, line width, and specific use of symbols described in other graphical style manuals were not considered in our graphical analysis data collection form.⁹, ¹¹, ¹², ¹³, ¹⁴ Readers and authors can acquire more information about important considerations in producing clear, high-quality graphics by examining texts on scientific graphics preparation.⁷, ⁸, ⁹, ¹¹, ¹², ¹³

Finally, we realize the apparent subjective nature of our review; this article sits somewhere between science and aesthetic criticism. Others may believe that detailed graphics are not required for scientific reporting but necessary only for data analysis. We disagree. Graphics are not decoration for text, just as medical journals are not advertisements for scientific results but a forum for the exchange of information among professionals through the publication of clear and detailed experimental results. Data should be presented in sufficient detail that each reader can critically evaluate the results and reach his or her own conclusion regarding the validity of the authors’ interpretation. With an ever-greater percentage of research being funded by groups vested in the study’s outcome, disclosure of detailed data becomes all the more important.

We performed this review to draw attention to an important feature of research and manuscript preparation. The best efforts in experimental planning and execution are for naught if detailed data are not presented in a clear, concise manner that highlights important themes. Researchers should expend the minimal effort required to take data that have been prepared for computerized statistical analysis, and create high-quality analytic and presentation graphics. We hope this review and the accompanying concepts article will stimulate authors to consider the importance of graphics to communicate scientific results, foster the education of individuals wanting to improve their graphical displays, and improve the graphical literacy of Annals of Emergency Medicine. ²³

Back to Article Outline

Acknowledgements 

We thank Edward Lin for his outstanding work designing the database and managing the articles and graphics, and Patrick Gibbons for measuring the data density index denominator on each graphic.

Back to Article Outline

References 

1. Anonymous . In: 16 ed.  Kaplan J editors. Bartlett’s Familiar Quotations. Boston, MA: : Little, Brown; 1992;p. 782
   • View In Article
2. Tufte ER. Visual Display of Quantitative Information. Cheshire, CT: : Graphic Press; 1983;
   • View In Article
3. Cleveland WS. Graphs in scientific publications. Am Stat. 1984;38:261–269
   • View In Article
4. De Amici A, Klersy C, Tinelli C. Graphic data representation in anaesthesiological journals: a proposed methodology for assessment of appropriateness. Anaesth Intens Care. 1997;25:659–664
   • View In Article
5. Wainer H. How to display data badly. Am Stat. 1984;38:137–147
   • View In Article
6. Tufte ER. Envisioning Information. Cheshire, CT: : Graphic Press; 1990;
   • View In Article
7. Tukey JW. The Collected Works of John Tukey. Vol V. Monterey, CA: : Wadsworth Advanced Books & Software; 1984; Graphics: 1965-1985
   • View In Article
8. Cleveland WS. Visualizing Data. Summit, NJ: : Hobart Press; 1993;
   • View In Article
9. Cleveland WS. Elements of Graphing Data. In: Monterey, CA: : Wadsworth Advanced Books and Software; 1985;p. 229–294
   • View In Article
10. Wainer H, Thissen D. Graphical data analysis. Annu Rev Psychol. 1981;32:192–241
    • View In Article
11. MacGregor AL. Graphics Simplified. How to Plan and Prepare Effective Charts, Graphs, Illustrations and Other Visual Aids. Toronto, Ontario, Canada: : University of Toronto Press; 1979;
    • View In Article
12. Briscoe MH. A Researcher’s Guide to Scientific and Medical Illustrations. In: New York, NY: : Springer-Verlag; 1990;p. 71–114 75-107
    • View In Article
13. Briscoe MH. Preparing Scientific Illustrations: A Guide to Better Posters, Presentations, and Publications. 2nd ed. New York, NY: : Springer-Verlag; 1996;
    • View In Article
14. Manuscript preparation . In: 9th ed.  Iverson C,  Flanagin A,  Fontanarosa PB, et al. editor. American Medical Association Manual of Style: A Guide for Authors and Editors. Baltimore, MD: : Williams & Wilkins; 1998;p. 7–85
    • View In Article
15. Scientific Illustration Committee . Illustrating Science: Standards for Publication. Bethesda, MD: : Council of Biology Editors; 1988;
    • View In Article
16. Cox DR. Some remarks on the role in statistics of graphical methods. Appl Statist. 1978;27:4–9
    • View In Article
17. McNeil D. On graphing paired data. Am Stat. 1992;46:307–311
    • View In Article
18. Tufte ER. Multifunctioning graphical elements. In:  Tufte ER editors. Visual Display of Quantitative Information. Cheshire, CT: : Graphic Press; 1983;p. 139–159
    • View In Article
19. Tufte ER. Data density and small multiples. In:  Tufte ER editors. Visual Display of Quantitative Information. Cheshire, CT: : Graphic Press; 1983;p. 170–175
    • View In Article
20. Tufte ER. Small multiples. In:  Tufte ER editors. Envisioning Information. Cheshire, CT: : Graphic Press; 1990;p. 67–80
    • View In Article
21. Tufte ER. Chartjunk: vibrations, grids, and ducks. In:  Tufte ER editors. Visual Display of Quantitative Information. Cheshire, CT: : Graphic Press; 1983;p. 107–121
    • View In Article
22. Tufte ER. Data density and small multiples. In:  Tufte ER editors. Visual Display of Quantitative Information. Cheshire, CT: : Graphic Press; 1983;p. 161–169
    • View In Article
23. Schriger DL, Cooper RJ. Achieving graphical excellence: suggestions and methods for creating high-quality visual displays of experimental data. Ann Emerg Med. 2001;37:75–87
    • View In Article
    • Abstract
    • Full Text
    • Full-Text PDF (396 KB)
    • CrossRef
24. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;8476:307–310
    • View In Article