| | The Content of Medical Journal Instructions for AuthorsPresented at the 5th International Congress on Peer Review and Biomedical Publication, Chicago, IL, September 16-18, 2005. Study 1 was presented orally and Study 2 as a poster. Received 23 January 2006; received in revised form 17 March 2006; accepted 20 March 2006. published online 05 June 2006. Study objectiveWe describe the general and statistical content of the Instructions for Authors of major medical journals. MethodsThis article reports on 2 observational studies. In study 1, we investigated the online versions of Instructions for Authors of 166 journals from 33 specialties for the presence of content about 15 methodologic and statistical topics. In study 2, we categorized the general content of the online versions of the Instructions for Authors of 35 medical journals. Two abstractors independently assigned the content into 18 categories and counted the total number of words devoted to each category. Interrater reliability of the classification was assessed. ResultsLess than half of the 166 Instructions for Authors in study 1 provided any guidance on statistical methods, and the majority failed to cite accepted publication standards such as the International Committee of Medical Journal Editors Uniform Guidelines or CONSORT. Only 13% of journals commented on the content and style of data tables and figures. The 35 Instructions for Authors in study 2 varied greatly in length (mean 3,308; median 2,283; range 885 to 18,927) and, with few exceptions, focused on formatting issues. Forty-three percent of Instructions offered no advice on scientific content, and only 5 journals devoted more than 10% of their words to scientific content. ConclusionThere is great heterogeneity among medical journal Instructions for Authors. Instructions provide little guidance about methodologic and statistical issues, and the advice provided is often contradictory among journals. Introduction  Background Publication guidelines such as CONSORT, STARD, QUOROM, and MOOSE were created to improve the quality of the medical literature.1, 2, 3, 4 These guidelines, which provide content recommendations for reports of randomized trials, studies of diagnostic tests, meta-analyses of randomized controlled trials, and meta-analyses of observational studies are not comprehensive; they are not intended to address all quality issues in medical publishing. Journals can use their Instructions for Authors to refer authors to these guidelines and to inform them of expectations about scientific issues not covered by the guidelines. In addition to providing advice about scientific issues, Instructions for Authors provide authors with information about the journal (its readership, its purpose, etc), its review process, the types of articles it publishes (and their organization and length limitations), and the formatting of submissions, though the last of these has diminished in importance in this age of electronic submission. Editor’s Capsule SummaryWhat is already known on this topic Journals use Instructions for Authors to communicate expectations about submitted articles. The degree of standardization among Instructions for Authors is unknown, as is what topics they provide as guidance to authors. What question this study addressed This study analyzed the methodologic-statistical and general content of Instructions for Authors of better-quality medical journals that publish original research. What this study adds to our knowledge There was great heterogeneity in the content and length of the 166 Instructions that were examined. Most Instructions provide little guidance about methodologic and statistical issues, despite their importance in assessing the quality of research. How this might change clinical practice Journal editors should examine their Instructions to determine whether they provide advice that is likely to optimize the scientific quality of the articles published in their journal. Importance There has been increased scrutiny of the medical literature of late because of concerns about conflict of interest and optimism bias.5, 6, 7 Publication guidelines such as CONSORT are designed to reduce the possibility of unrecognized bias by providing readers with the information they need to determine whether such bias exists. There is some evidence that they work.8, 9 There is no evidence that better instructions for authors produce better articles, and, given the number of submissions that fail to follow the instructions, some editors and peer reviewers may rightly wonder how often authors actually read them. We believe, however, that the logic of the CONSORT effort—improve the completeness of the reporting and the validity of the conclusions will improve—applies to efforts to improve instructions for authors. Furthermore, it is likely that a journal that revises and improves its Instructions for Authors benefits in ways that go beyond the effect on the authors who read them. The act of revising the Instructions likely helps the journal clarify the goals of its peer review and editorial processes. Goals of This Investigation We carried out this research to identify the amount and type of methodologic and statistical advice offered in the Instructions for Authors of medical journals that publish original research and to characterize the general content of these Instructions. We believed that we would find considerable heterogeneity. Our goal was descriptive. Our aim was to provide evidence that would stimulate journals to improve the completeness of their Instructions and, by extension, perhaps, the quality and consistency of the articles they publish. Materials and methods Our study consists of 2 separate but related efforts that were carried out sequentially. The first addressed the presence or absence of specific methodologic and statistical content in the Instructions for Authors; the second examined and categorized the overall content. For both studies, we used journal impact factors calculated by the Institute for Scientific Information (ISI), an imperfect but widely used method for rating journals, to identify the top journals as defined by the ISI. 10 We selected this group so that we would characterize the better and more widely read journals in each field. For the study of methodologic and statistical advice, we studied the ISI 2001 (the most recent rankings available at the time of the study) top 5 journals publishing original research in all 33 clinical categories, as well as the top 15 general and internal medicine journals, a total of 166 journals, which is less than 180 (5×33+15) because some journals are listed in more than 1 category. For the general content study, we randomly selected 10 of the 33 clinical categories because we wished to limit the number of Instructions we needed to score. We used the 2002 ISI impact factors and chose the first- and fifth-ranked journals in these 10 clinical categories and the top 15 ranked general and internal medicine journals, a total of 35 journals. Randomization was done with the uniform function of Stata, version 8.0 (Stata Corporation, College Station, TX). General Content Study We downloaded the online version of the selected journals’ Instructions for Authors between July and November 2004 and used an algorithm created specifically for this project to parse every sentence into one of 18 categories grouped into 4 major classes: general information, specific information, formatting, and scientific content (Table 2, algorithm available as Appendix E1 available online at http://www.annemergmed.com). We created this algorithm after failing to find any suitable instrument for this purpose in the literature. Two authors (SA,DLS) independently classified the first 6 Instructions to pilot and refine the algorithm and the next 6 to formally assess interrater reliability. We categorized all material on each journal’s Instructions for Authors Web page except forms (eg, copyright, conflict of interest, author’s checklist) and linked pages outside the site (eg, a link to the International Committee of Medical Journal Editors (ICMJE) Web site). | | |  | Category/Subcategory | No. (%) of Instructions With Item | Mean Words | All Words, % | |
|---|
| All Journals, n=35 | <2000 Words, n=15 | 2000-9999, n=18 | 10,000+ Words, n=2 | |
|---|
| Format | 35 100 | 1733 | 52 | 67 | 62 | 24 | | | Abstract format | 3497 | 146 | 4 | 5 | 5 | 3 | | | Reference style | 3291 | 212 | 6 | 10 | 7 | 2 | | | Format (how to format submission) | 3497 | 446 | 13 | 19 | 17 | 4 | | | Logistics (how to submit…) | 35100 | 929 | 28 | 33 | 33 | 15 | | | General information | 35100 | 538 | 16 | 16 | 15 | 18 | | | General information (about the journal…) | 3497 | 123 | 4 | 7 | 3 | 2 | | | Types of articles the journal publishes | 2983 | 415 | 13 | 9 | 12 | 15 | | | Specific information | 3497 | 686 | 21 | 15 | 19 | 29 | | | Journal’s right to access data | 823 | 8 | 0 | 0 | 0 | 0 | | | Journal’s use of World Wide Web | 1029 | 52 | 2 | 0 | 1 | 3 | | | Handling the media (embargo date…) | 1029 | 57 | 2 | 1 | 1 | 4 | | | Duplicate publication | 2983 | 71 | 2 | 2 | 2 | 3 | | | Reference to guideline (CONSORT…) | 2880 | 87 | 3 | 2 | 3 | 4 | | | Authorship (definition…) | 2674 | 93 | 3 | 2 | 3 | 4 | | | Peer-review process | 2366 | 97 | 3 | 1 | 2 | 6 | | | Human/animal subjects | 2777 | 97 | 3 | 2 | 3 | 2 | | | Conflict of interest | 2880 | 123 | 4 | 2 | 4 | 5 | | | Scientific content | 2057 | 351 | 11 | 3 | 4 | 30 | | | Content of the tables and figures | 1029 | 15 | 0 | 0 | 0 | 1 | | | Content of the methods | 1440 | 129 | 4 | 1 | 2 | 10 | | | Content, general | 1234 | 207 | 6 | 1 | 2 | 18 | | | Total | | 3308 | 100 | 100 | 100 | 100 | | | | |
We analyzed our findings by examining the number of words in each category and the percentage of journals that made any mention of that category. Per our protocol, we stratified journals by journal type (general versus specialty), and rank (first versus fifth) because we thought that differences among these strata would be interesting, though we had no hypotheses about what differences we might find. We also stratified on overall length to examine whether longer Instructions contained more material on scientific content. Results Methodological and Statistical Content Study Less than half of the 166 journals provided information on statistical methods (Table 1). Eighty-seven percent (13/15) of general journals and 36% (54/151) of specialty journals made reference to ICMJE uniform requirements. 11 Fifty-three percent (8/15) of general journals and 19% (29/151) of specialty journals cited CONSORT. 11 All other methodologic and statistical categories were addressed in less than 40% of the Instructions in the sample. General journals were more likely to refer to reporting guidelines but rather less likely to include advice about statistical issues (Table 1). Few journals (13%) commented on the use of tables and figures. Instructions that referenced methodology articles cited from 1 to 49 (median, 1; interquartile ratio 1, 5). We found no evidence of a predominant pattern of advice or a core set of standard advice that was provided by most journals. Most Instructions offered no advice about P values and CIs. The advice provided by those that did was often contradictory, eg, “Report actual P values, rather than ranges or limits” versus “Statistical probability (P) should be reported … at only one of the following levels P < 0.05, 0.01, 0.005, and 0.001.” Similar contradictions were observed for other topics: “Tables usually convey more precise numerical information; graphs should be reserved for highlighting changes over time or between treatments” versus “For presentation of data, figures are preferred to tables.” General Content Study Ninety-five percent (95% CI 94% to 96%) of the words contained in the 6 sets of Instructions for Authors that were subjected to interrater reliability testing were placed in the same category by both raters. There was great heterogeneity of the total length of the Instructions (mean words 3,308; median 2,283; range 885 to 18,927) (Figure 1). Although all Instructions provided general information about the journal and about the formatting of submissions, only 57% addressed issues related to the scientific content of the article, and only 11% of all words were devoted to scientific content (Table 2). The 2 journals with lengthy (>10,000 words) Instructions (British Medical Journal and Annals of Emergency Medicine) devoted 30% of their words to scientific content, whereas the others devoted less than 4% on average (Table 2 and Figure 2). Longer Instructions addressed more of the 18 content areas (Figure 3). The content of the Instructions did not vary appreciably between general and specialty journals (Figure 3 and Figure E1, available online at http://www.annemergmed.com) or rank (Figure E2, Figure E3, available online at http://www.annemergmed.com). Limitations The Instructions for Authors included in these samples were based on the ISI impact factor, which is an imperfect mechanism for measuring journal impact or quality. 12 In addition, journals frequently change their Instructions for Authors, and current Instructions may be better than those in our samples, which were obtained in 2003 and 2004. On the other hand, we sampled from higher-ranked journals, so we may have biased estimates of items that are associated with ISI rank. Both studies depend on the reasonableness of their classification taxonomies. We could not find any validated instruments for this purpose and developed these specifically for these studies. Although they are not validated, we believe their content validity is evident. In the study of general content, we used the percentage of words devoted to a topic as a proxy for the amount of content on that topic, yet the numbers of words is an imperfect proxy for the number of thoughts. Though it is possible that some journals keep their Instructions short by referring to external content that we did not count, Instructions that referred to CONSORT were longer than those that did not (median words 2,380 versus 1,835). Furthermore, by measuring the number of journals that had any content on each of the 18 topics, we avoided some of the problems inherent in using word count. Because both methods produced similar findings, we are comfortable that word count and the use of percentages based on it, although imperfect, provide a reasonable proxy for content. Discussion Our investigations demonstrate that journals’ Instructions for Authors are heterogeneous in their length, that a small portion of Instructions for Authors’ space is devoted to scientific content, and that methodologic and statistical advice, when offered, is variable and sometimes contradictory. By documenting this heterogeneity, we hope to stimulate individual journals to examine their Instructions and consider how they might be improved to better inform authors how to write an article that the journal considers to be of good scientific quality. We also hope to stimulate organizations such as the ICMJE and the Council of Science Editors to standardize advice about methodologic and statistical issues for topics not covered by existing guidelines such as CONSORT. Our study was not designed to explain the heterogeneity we encountered, and for some of our findings, we are unable to offer reasonable postulates. For example, we cannot understand why 59% of journals would fail to refer to the ICMJE Uniform Requirements and 78% of journals would fail to mention CONSORT. Is it possible that these journals are unaware of these guidelines? We also are at a loss to fully explain why so few journals provide advice related to methodologic and statistical issues. Despite the existence of several general guidelines on the use of statistics, 13, 14, 15 most journals have chosen, either actively or by default, to avoid providing guidance on issues related to the scientific content of the articles they publish. There are several plausible explanations for this phenomenon. First, journals may not care to regulate this aspect of their publication or trust the peer review process to sort it all out or do not believe that authors read the Instructions for Authors. Second, journals may be resistant to take sides in the controversy over what constitutes best statistical practice. Third, journal editors may have an opinion about best methodologic or statistical practice but refrain from stating their wishes out of fear that any attempt at regulation may deter authors from submitting to their journal. Fourth, some journals may have completely overlooked this aspect of their publication process. It remains unclear whether standardization is desirable or possible and, if achieved, how it would affect the quality of published medical research. Standardization would certainly be of benefit to authors, who could submit their work to any journal, knowing that they need only comply with one set of Instructions that they could better familiarize themselves with. For example, would it not be easier if all journals had the same policy about P values and CIs? In summary, we found little of evidence of standardization or consistency among medical journal instructions for authors. Instructions typically contain little advice on matters related to the scientific quality of research publications, focusing instead on the formatting of submissions. The authors thank Varda Appleton Schriger for help with the downloading of Web site content and the pilot testing of the classification taxonomy and Vladislav J. Mikulich for help with development of the graphics. Appendix E1. Algorithm for classification of text in the study of general content. The general principle is to choose the most specific category. The algorithm therefore starts by examining whether each phrase fits into a niche category. If it does not, the rater proceeds to more general categories. Does the phrase pertain to | 1.Anything about the content or sections of the abstract | If yes, score ABSTRACT | | | 2.Anything about the style or content of the references | If yes, score REFERENCES | | | 3.Duplicate publication | If yes, score DUP PUB | | | 4.Human or animal subjects | If yes, score IRB | | | 5.Handling the media or embargo dates | If yes, score MEDIA | | | 6.Definition of authorship, indicating who did what | If yes, score AUTHORSHIP | | | 7.Journal right to access data (journal can request data for peer review) | If yes, score DATA | | | 8.Journal’s use of the WWW (online only articles, online supplements…) | If yes score WWW | | | 9.Conflict of interest issues (author, reviewer, editor, sponsor…) | If yes, score COI | | | 10.Journal’s peer review process (timing, how…) | If yes, score PEER REVIEW | | | 11.Any references to publication guidelines (CONSORT, MOOSE, QUOROM, STARD, ICMJE Uniform Requirements…)? | If yes, score GUIDELINE | | | 12.Material about the types of articles the journals publishes (includes word limits and characteristics of each type) | If yes, score ART. TYPE | | | If none of the above: | | | | Decide if phrase is information or instructions about scientific content | | | | | | | If content: | | | | 13.Is it about the content or style (but not the format [eg, “put each table on a separate page or use a true type 12 point font” or “submit as epg or jpg file”) of tables and figures? | If yes, score T & F | | | 14.Is it about methodology or statistical issues? | If yes, score METH/STAT | | | | If not, score GEN. CONTENT (15) (which includes nonstatistical instructions such as “divide your paper into the following sections…,” and “avoid statements such as ‘this is the first paper to examine…’”) | | | | | | | If information: | | | | 15.Is it about logistics (where to send things, what file types are accepted (eg, “submits graphs as epg files,” copyright, page costs, reprints, permissions for reprinting, instructions for e-submission)? | If yes, score LOGISTICS | | | 16.Is it about format (eg, “double space all sections,” “begin each section on a separate page,” title page information, drug names, units of measurement, abbreviations)? | If yes, score FORMAT | | | If not, score as GEN. INFO. (18) (which includes such items as who owns the journal, who are the editor and managing editor, what its overall goals are, what it wants to publish, what the circulation is….) | | | | |
References 1. 1Moher D, Schulz KF, Altman DGCONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357:1191–1194. Abstract | Full Text |
Full-Text PDF (75 KB)
|
CrossRef
2. 2Bossuyt PM, Reitsma JB, Bruns DE, et al. Standards for reporting of diagnostic accuracy: towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative: Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003;49:1–6. MEDLINE |
CrossRef
3. 3Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement: quality of reporting of meta-analyses. Lancet. 1999;354:1896–1900. Abstract | Full Text |
Full-Text PDF (86 KB)
|
CrossRef
4. 4Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting: Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012. MEDLINE |
CrossRef
5. 5Horton R. The Dawn of McScience. New York, NY: Rev Books; 2004;. 6. 6Smith R. Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med. 2005;2:e138.
CrossRef
7. 7Chalmers I, Matthews R. What are the implications of optimism bias in clinical research?. Lancet. 2006;367:449–450. Full Text |
Full-Text PDF (92 KB)
|
CrossRef
8. 8Plint AC, Moher D, Schulz K, et al. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? (a systematic review). 2005;. 9. 9Moher D, Jones A, Lepage LCONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285:1992–1995. MEDLINE |
CrossRef
10. 10Thomson Scientific. The ISI impact factor. 2006;. 11. 11Altman DG. Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors. BMJ. 2005;330:1056–1057. 12. 12Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90–93.
CrossRef
13. 13Lang TA, Secic M. How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. Philadelphia, PA: American Collage of Physicians; 1997;. 14. 14Altman DG, Gore SM, Gardner MJ, et al. Statistical guidelines for contributors to medical journals. In: Altman DG, Machin D, Bryant TN editor. Statistics with Confidence: Confidence Intervals and Statistical Guidelines. London: BMJ Books; 2000;p. 171–190. 15. 15Bailar JC, Mosteller F. Guidelines for statistical reporting in articles for medical journals: amplifications and explanations. Ann Intern Med. 1988;108:266–273. MEDLINE a University of California, Los Angeles Emergency Medicine Center, University of California, Los Angeles School of Medicine, Los Angeles, CA b Centre for Statistics in Medicine, Oxford, UK.  Address for correspondence: David L. Schriger, MD, MPH, 924 Westwood Blvd. # 300, Los Angeles, CA 90024-2924; 310-794-0593, fax 310-794-0599.
Supervising editor: Michael L. Callaham, MD Author contributions: DGA conceived and designed the methodological and statistical content study and did the initial data abstraction. DLS and DGA conceived and designed general content study 2. DLS and SA did the data collection. DLS and SA did the data analysis. DLS checked the abstractions and analyzed the data. DLS drafted the manuscript, which was edited by DGA and SA. DLS takes responsibility for the paper as a whole. Funding and support: The authors report this study did not receive any outside funding or support. Reprints not available from the authors. PII: S0196-0644(06)00478-1 doi:10.1016/j.annemergmed.2006.03.028 © 2006 American College of Emergency Physicians. Published by Elsevier Inc. All rights reserved. | |
|
FULL TEXT00478-1/journalimage?img=PIIS0196064406004781.gr1.lrg.gif&fig=fig1&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)
00478-1/journalimage?img=PIIS0196064406004781.gr2.lrg.gif&fig=fig2&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)
00478-1/journalimage?img=PIIS0196064406004781.gr3.lrg.gif&fig=fig3&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)
00478-1/journalimage?img=PIIS0196064406004781.gre1.lrg.gif&fig=fige1&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)
00478-1/journalimage?img=PIIS0196064406004781.gre2.lrg.gif&fig=fige2&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)
00478-1/journalimage?img=PIIS0196064406004781.gre3.lrg.gif&fig=fige3&kwhquery=&issn=0196-0644&ishighres=false&allhighres=false&free=yes','journalimage');)