The Content of Medical Journal Instructions for Authors

Article Outline

• Abstract
• Introduction
  • Background
  • Importance
  • Goals of This Investigation
• Materials and methods
  • Methodological and Statistical Content Study
  • General Content Study
• Results
  • Methodological and Statistical Content Study
  • General Content Study
• Limitations
• Discussion
• Acknowledgment
• Appendix E1. 
• References
• Copyright

Study objective

We describe the general and statistical content of the Instructions for Authors of major medical journals.

Methods

This article reports on 2 observational studies. In study 1, we investigated the online versions of Instructions for Authors of 166 journals from 33 specialties for the presence of content about 15 methodologic and statistical topics. In study 2, we categorized the general content of the online versions of the Instructions for Authors of 35 medical journals. Two abstractors independently assigned the content into 18 categories and counted the total number of words devoted to each category. Interrater reliability of the classification was assessed.

Results

Less than half of the 166 Instructions for Authors in study 1 provided any guidance on statistical methods, and the majority failed to cite accepted publication standards such as the International Committee of Medical Journal Editors Uniform Guidelines or CONSORT. Only 13% of journals commented on the content and style of data tables and figures. The 35 Instructions for Authors in study 2 varied greatly in length (mean 3,308; median 2,283; range 885 to 18,927) and, with few exceptions, focused on formatting issues. Forty-three percent of Instructions offered no advice on scientific content, and only 5 journals devoted more than 10% of their words to scientific content.

Conclusion

There is great heterogeneity among medical journal Instructions for Authors. Instructions provide little guidance about methodologic and statistical issues, and the advice provided is often contradictory among journals.

Back to Article Outline

Introduction 

Background 

Publication guidelines such as CONSORT, STARD, QUOROM, and MOOSE were created to improve the quality of the medical literature.¹, ², ³, ⁴ These guidelines, which provide content recommendations for reports of randomized trials, studies of diagnostic tests, meta-analyses of randomized controlled trials, and meta-analyses of observational studies are not comprehensive; they are not intended to address all quality issues in medical publishing. Journals can use their Instructions for Authors to refer authors to these guidelines and to inform them of expectations about scientific issues not covered by the guidelines.

In addition to providing advice about scientific issues, Instructions for Authors provide authors with information about the journal (its readership, its purpose, etc), its review process, the types of articles it publishes (and their organization and length limitations), and the formatting of submissions, though the last of these has diminished in importance in this age of electronic submission.

Importance 

There has been increased scrutiny of the medical literature of late because of concerns about conflict of interest and optimism bias.⁵, ⁶, ⁷ Publication guidelines such as CONSORT are designed to reduce the possibility of unrecognized bias by providing readers with the information they need to determine whether such bias exists. There is some evidence that they work.⁸, ⁹ There is no evidence that better instructions for authors produce better articles, and, given the number of submissions that fail to follow the instructions, some editors and peer reviewers may rightly wonder how often authors actually read them. We believe, however, that the logic of the CONSORT effort—improve the completeness of the reporting and the validity of the conclusions will improve—applies to efforts to improve instructions for authors. Furthermore, it is likely that a journal that revises and improves its Instructions for Authors benefits in ways that go beyond the effect on the authors who read them. The act of revising the Instructions likely helps the journal clarify the goals of its peer review and editorial processes.

Goals of This Investigation 

We carried out this research to identify the amount and type of methodologic and statistical advice offered in the Instructions for Authors of medical journals that publish original research and to characterize the general content of these Instructions. We believed that we would find considerable heterogeneity. Our goal was descriptive. Our aim was to provide evidence that would stimulate journals to improve the completeness of their Instructions and, by extension, perhaps, the quality and consistency of the articles they publish.

Back to Article Outline

Materials and methods 

Our study consists of 2 separate but related efforts that were carried out sequentially. The first addressed the presence or absence of specific methodologic and statistical content in the Instructions for Authors; the second examined and categorized the overall content. For both studies, we used journal impact factors calculated by the Institute for Scientific Information (ISI), an imperfect but widely used method for rating journals, to identify the top journals as defined by the ISI. ¹⁰ We selected this group so that we would characterize the better and more widely read journals in each field.

For the study of methodologic and statistical advice, we studied the ISI 2001 (the most recent rankings available at the time of the study) top 5 journals publishing original research in all 33 clinical categories, as well as the top 15 general and internal medicine journals, a total of 166 journals, which is less than 180 (5×33+15) because some journals are listed in more than 1 category. For the general content study, we randomly selected 10 of the 33 clinical categories because we wished to limit the number of Instructions we needed to score. We used the 2002 ISI impact factors and chose the first- and fifth-ranked journals in these 10 clinical categories and the top 15 ranked general and internal medicine journals, a total of 35 journals. Randomization was done with the uniform function of Stata, version 8.0 (Stata Corporation, College Station, TX).

Methodological and Statistical Content Study 

One author (DGA) obtained the online Instructions for Authors for each journal between January and May 2003 and placed all material related to methodology and statistics in a text file. ¹¹ After failing to find an existing taxonomy for this purpose, we created a 15-item taxonomy for classifying instructions about methodologic and statistical issues. This taxonomy was constructed by our knowledge of the content of several journals’ Instructions for Authors and our desire to learn whether Instructions for Authors provided advice about topics that we thought were important. We then scored the methodologic and statistical comments from each journal to determine whether they addressed each of the 15 items (Table 1), which was accomplished by reading each comment and by using text searches for relevant words (eg, searching on “confidence” and “CI” to find all mentions of confidence interval [CI]). The categorization of all statements in all Instructions that had at least 1 methodologic or statistical comment was checked by a second author (DLS). Discrepancies were adjudicated by these 2 authors. Data are presented as the percentage of the 166 journals that made mention of each category, with 95% CIs for these percentages. All analyses were carried out in Stata version 8.0.

Table 1. Percentage of 166 journals that addressed each of 15 methodologic and statistical categories in the Instructions for Authors, stratified by journal type

What the Instructions Address	Percentage of Instructions With This Item (95% CI)
	All Journals, n=166	General, n=15	Specialty, n=151	Difference
ICJME Uniform Requirements	41(33-49)	87	36	50(24-62)
CONSORT	22(16-29)	53	19	35(11-57)
Other guidelines (QUOROM, STARD…)	4(2-8)	33	1	32(14-57)
Provide enough detail for reproducibility	11(7-17)	0	12	−12(−18To9)
Anything about statistical methods	39(32-47)	27	40	−14(−31To13)
P values	20(15-27)	20	21	−1(−15To25)
CIs	14(9-21)	7	15	−9(−17To15)
Multiple testing	5(3-10)	0	6	−6(−11To15)
Modeling	4(2-8)	0	5	−5(−9To16)
Power	10(6-16)	0	11	−11(−17To10)
Sensitivity analysis	4(2-8)	0	5	−5(−18To10)
Bayesian methods	1(0-4)	7	1	6(0-29)
Tables and figures	13(8-19)	7	13	−7(−20To7)
Statistical consultants	6(3-11)	7	6	1(−7To24)
Reference methods/statistics articles	10(6-16)	33	8	25(6-51)

General Content Study 

We downloaded the online version of the selected journals’ Instructions for Authors between July and November 2004 and used an algorithm created specifically for this project to parse every sentence into one of 18 categories grouped into 4 major classes: general information, specific information, formatting, and scientific content (Table 2, algorithm available as Appendix E1 available online at http://www.annemergmed.com). We created this algorithm after failing to find any suitable instrument for this purpose in the literature. Two authors (SA,DLS) independently classified the first 6 Instructions to pilot and refine the algorithm and the next 6 to formally assess interrater reliability. We categorized all material on each journal’s Instructions for Authors Web page except forms (eg, copyright, conflict of interest, author’s checklist) and linked pages outside the site (eg, a link to the International Committee of Medical Journal Editors (ICMJE) Web site).

Table 2. Presence of categories, mean words per category, and percentage of words per category for 35 journals’ Instructions for Authors, stratified by total word count

Category/Subcategory	No. (%) of Instructions With Item	Mean Words	All Words, %
			All Journals, n=35	<2000 Words, n=15	2000-9999, n=18	10,000+ Words, n=2
Format	35100	1733	52	67	62	24
Abstract format	3497	146	4	5	5	3
Reference style	3291	212	6	10	7	2
Format (how to format submission)	3497	446	13	19	17	4
Logistics (how to submit…)	35100	929	28	33	33	15
General information	35100	538	16	16	15	18
General information (about the journal…)	3497	123	4	7	3	2
Types of articles the journal publishes	2983	415	13	9	12	15
Specific information	3497	686	21	15	19	29
Journal’s right to access data	823	8	0	0	0	0
Journal’s use of World Wide Web	1029	52	2	0	1	3
Handling the media (embargo date…)	1029	57	2	1	1	4
Duplicate publication	2983	71	2	2	2	3
Reference to guideline (CONSORT…)	2880	87	3	2	3	4
Authorship (definition…)	2674	93	3	2	3	4
Peer-review process	2366	97	3	1	2	6
Human/animal subjects	2777	97	3	2	3	2
Conflict of interest	2880	123	4	2	4	5
Scientific content	2057	351	11	3	4	30
Content of the tables and figures	1029	15	0	0	0	1
Content of the methods	1440	129	4	1	2	10
Content, general	1234	207	6	1	2	18
Total		3308	100	100	100	100

We analyzed our findings by examining the number of words in each category and the percentage of journals that made any mention of that category. Per our protocol, we stratified journals by journal type (general versus specialty), and rank (first versus fifth) because we thought that differences among these strata would be interesting, though we had no hypotheses about what differences we might find. We also stratified on overall length to examine whether longer Instructions contained more material on scientific content.

Back to Article Outline

Results 

Methodological and Statistical Content Study 

Less than half of the 166 journals provided information on statistical methods (Table 1). Eighty-seven percent (13/15) of general journals and 36% (54/151) of specialty journals made reference to ICMJE uniform requirements. ¹¹ Fifty-three percent (8/15) of general journals and 19% (29/151) of specialty journals cited CONSORT. ¹¹ All other methodologic and statistical categories were addressed in less than 40% of the Instructions in the sample. General journals were more likely to refer to reporting guidelines but rather less likely to include advice about statistical issues (Table 1). Few journals (13%) commented on the use of tables and figures. Instructions that referenced methodology articles cited from 1 to 49 (median, 1; interquartile ratio 1, 5). We found no evidence of a predominant pattern of advice or a core set of standard advice that was provided by most journals.

Most Instructions offered no advice about P values and CIs. The advice provided by those that did was often contradictory, eg, “Report actual P values, rather than ranges or limits” versus “Statistical probability (P) should be reported … at only one of the following levels P < 0.05, 0.01, 0.005, and 0.001.” Similar contradictions were observed for other topics: “Tables usually convey more precise numerical information; graphs should be reserved for highlighting changes over time or between treatments” versus “For presentation of data, figures are preferred to tables.”

General Content Study 

Ninety-five percent (95% CI 94% to 96%) of the words contained in the 6 sets of Instructions for Authors that were subjected to interrater reliability testing were placed in the same category by both raters. There was great heterogeneity of the total length of the Instructions (mean words 3,308; median 2,283; range 885 to 18,927) (Figure 1). Although all Instructions provided general information about the journal and about the formatting of submissions, only 57% addressed issues related to the scientific content of the article, and only 11% of all words were devoted to scientific content (Table 2). The 2 journals with lengthy (>10,000 words) Instructions (British Medical Journal and Annals of Emergency Medicine) devoted 30% of their words to scientific content, whereas the others devoted less than 4% on average (Table 2 and Figure 2). Longer Instructions addressed more of the 18 content areas (Figure 3). The content of the Instructions did not vary appreciably between general and specialty journals (Figure 3 and Figure E1, available online at http://www.annemergmed.com) or rank (Figure E2, Figure E3, available online at http://www.annemergmed.com).

• 
  • View Large Image
  • Download to PowerPoint

• Figure 1. 

  The number of words devoted to the 4 major categories by each journal. Note how few words are devoted to scientific content. See Table 1 for details about the 18 subcategories.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 2. 

  The number of words devoted to the 4 major categories, stratified by the overall length of the Instructions. Only the longest journals have substantial scientific content.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 3. 

  This scatterplot demonstrates that the longer Instructions for Authors tend to address more of the 18 topic categories. General and specialty journals appear to be on the same curve.

• 
  • View Large Image
  • Download to PowerPoint

• Figure E1. 

  The number of words devoted to the 4 major categories, stratified by journal status (specialty vs general). There is little difference between the 2 groups.

• 
  • View Large Image
  • Download to PowerPoint

• Figure E2. 

  The length of the Instructions for Authors of the first- and fifth-ranked journals in 10 specialties is compared. Length does not systematically vary with journal ISI rank.

• 
  • View Large Image
  • Download to PowerPoint

• Figure E3. 

  This scatterplot demonstrates that, although longer Instructions for Authors tend to address more of the 18 topic categories, specialty journal rank is not associated with length or completeness in any important way.

Back to Article Outline

Limitations 

The Instructions for Authors included in these samples were based on the ISI impact factor, which is an imperfect mechanism for measuring journal impact or quality. ¹² In addition, journals frequently change their Instructions for Authors, and current Instructions may be better than those in our samples, which were obtained in 2003 and 2004. On the other hand, we sampled from higher-ranked journals, so we may have biased estimates of items that are associated with ISI rank.

Both studies depend on the reasonableness of their classification taxonomies. We could not find any validated instruments for this purpose and developed these specifically for these studies. Although they are not validated, we believe their content validity is evident.

In the study of general content, we used the percentage of words devoted to a topic as a proxy for the amount of content on that topic, yet the numbers of words is an imperfect proxy for the number of thoughts. Though it is possible that some journals keep their Instructions short by referring to external content that we did not count, Instructions that referred to CONSORT were longer than those that did not (median words 2,380 versus 1,835). Furthermore, by measuring the number of journals that had any content on each of the 18 topics, we avoided some of the problems inherent in using word count. Because both methods produced similar findings, we are comfortable that word count and the use of percentages based on it, although imperfect, provide a reasonable proxy for content.

Back to Article Outline

Discussion 

Our investigations demonstrate that journals’ Instructions for Authors are heterogeneous in their length, that a small portion of Instructions for Authors’ space is devoted to scientific content, and that methodologic and statistical advice, when offered, is variable and sometimes contradictory. By documenting this heterogeneity, we hope to stimulate individual journals to examine their Instructions and consider how they might be improved to better inform authors how to write an article that the journal considers to be of good scientific quality. We also hope to stimulate organizations such as the ICMJE and the Council of Science Editors to standardize advice about methodologic and statistical issues for topics not covered by existing guidelines such as CONSORT.

Our study was not designed to explain the heterogeneity we encountered, and for some of our findings, we are unable to offer reasonable postulates. For example, we cannot understand why 59% of journals would fail to refer to the ICMJE Uniform Requirements and 78% of journals would fail to mention CONSORT. Is it possible that these journals are unaware of these guidelines?

We also are at a loss to fully explain why so few journals provide advice related to methodologic and statistical issues. Despite the existence of several general guidelines on the use of statistics, ¹³, ¹⁴, ¹⁵ most journals have chosen, either actively or by default, to avoid providing guidance on issues related to the scientific content of the articles they publish. There are several plausible explanations for this phenomenon. First, journals may not care to regulate this aspect of their publication or trust the peer review process to sort it all out or do not believe that authors read the Instructions for Authors. Second, journals may be resistant to take sides in the controversy over what constitutes best statistical practice. Third, journal editors may have an opinion about best methodologic or statistical practice but refrain from stating their wishes out of fear that any attempt at regulation may deter authors from submitting to their journal. Fourth, some journals may have completely overlooked this aspect of their publication process.

It remains unclear whether standardization is desirable or possible and, if achieved, how it would affect the quality of published medical research. Standardization would certainly be of benefit to authors, who could submit their work to any journal, knowing that they need only comply with one set of Instructions that they could better familiarize themselves with. For example, would it not be easier if all journals had the same policy about P values and CIs?

In summary, we found little of evidence of standardization or consistency among medical journal instructions for authors. Instructions typically contain little advice on matters related to the scientific quality of research publications, focusing instead on the formatting of submissions.

Back to Article Outline

The authors thank Varda Appleton Schriger for help with the downloading of Web site content and the pilot testing of the classification taxonomy and Vladislav J. Mikulich for help with development of the graphics.

Back to Article Outline

Appendix E1. 

Algorithm for classification of text in the study of general content.

The general principle is to choose the most specific category. The algorithm therefore starts by examining whether each phrase fits into a niche category. If it does not, the rater proceeds to more general categories.

Does the phrase pertain to

1.Anything about the content or sections of the abstract	If yes, score ABSTRACT
2.Anything about the style or content of the references	If yes, score REFERENCES
3.Duplicate publication	If yes, score DUP PUB
4.Human or animal subjects	If yes, score IRB
5.Handling the media or embargo dates	If yes, score MEDIA
6.Definition of authorship, indicating who did what	If yes, score AUTHORSHIP
7.Journal right to access data (journal can request data for peer review)	If yes, score DATA
8.Journal’s use of the WWW (online only articles, online supplements…)	If yes score WWW
9.Conflict of interest issues (author, reviewer, editor, sponsor…)	If yes, score COI
10.Journal’s peer review process (timing, how…)	If yes, score PEER REVIEW
11.Any references to publication guidelines (CONSORT, MOOSE, QUOROM, STARD, ICMJE Uniform Requirements…)?	If yes, score GUIDELINE
12.Material about the types of articles the journals publishes (includes word limits and characteristics of each type)	If yes, score ART. TYPE
If none of the above:
Decide if phrase is information or instructions about scientific content
If content:
13.Is it about the content or style (but not the format [eg, “put each table on a separate page or use a true type 12 point font” or “submit as epg or jpg file”) of tables and figures?	If yes, score T & F
14.Is it about methodology or statistical issues?	If yes, score METH/STAT
	If not, score GEN. CONTENT (15) (which includes nonstatistical instructions such as “divide your paper into the following sections…,” and “avoid statements such as ‘this is the first paper to examine…’”)
If information:
15.Is it about logistics (where to send things, what file types are accepted (eg, “submits graphs as epg files,” copyright, page costs, reprints, permissions for reprinting, instructions for e-submission)?	If yes, score LOGISTICS
16.Is it about format (eg, “double space all sections,” “begin each section on a separate page,” title page information, drug names, units of measurement, abbreviations)?	If yes, score FORMAT
	If not, score as GEN. INFO. (18) (which includes such items as who owns the journal, who are the editor and managing editor, what its overall goals are, what it wants to publish, what the circulation is….)

Back to Article Outline

References 

1. Moher D, Schulz KF, Altman DG CONSORT Group. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357:1191–1194
   • View In Article
   • Abstract
   • Full Text
   • Full-Text PDF (75 KB)
   • CrossRef
2. Bossuyt PM, Reitsma JB, Bruns DE, et al. Standards for reporting of diagnostic accuracy: towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative: Standards for Reporting of Diagnostic Accuracy. Clin Chem. 2003;49:1–6
   • View In Article
   • MEDLINE
   • CrossRef
3. Moher D, Cook DJ, Eastwood S, et al. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement: quality of reporting of meta-analyses. Lancet. 1999;354:1896–1900
   • View In Article
   • Abstract
   • Full Text
   • Full-Text PDF (86 KB)
   • CrossRef
4. Stroup DF, Berlin JA, Morton SC, et al. Meta-analysis of observational studies in epidemiology: a proposal for reporting: Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA. 2000;283:2008–2012
   • View In Article
   • MEDLINE
   • CrossRef
5. Horton R. The Dawn of McScience. New York, NY: Rev Books; 2004;7-9
   • View In Article
6. Smith R. Medical journals are an extension of the marketing arm of pharmaceutical companies. PLoS Med. 2005;2:e138
   • View In Article
   • CrossRef
7. Chalmers I, Matthews R. What are the implications of optimism bias in clinical research?. Lancet. 2006;367:449–450
   • View In Article
   • Full Text
   • Full-Text PDF (92 KB)
   • CrossRef
8. Plint AC, Moher D, Schulz K, et al. Does the CONSORT checklist improve the quality of reports of randomized controlled trials? (a systematic review). 2005;Presented at the Fifth International Congress on Peer Review and Biomedical Publication, Chicago, IL, September 16-18
   • View In Article
9. Moher D, Jones A, Lepage L CONSORT Group (Consolidated Standards for Reporting of Trials). Use of the CONSORT statement and quality of reports of randomized trials: a comparative before-and-after evaluation. JAMA. 2001;285:1992–1995
   • View In Article
   • MEDLINE
   • CrossRef
10. Thomson Scientific. The ISI impact factor. 2006;Available at: http://scientific.thomson.com/free/essays/journalcitationreports/impactfactor. Accessed March 11
    • View In Article
11. Altman DG. Endorsement of the CONSORT statement by high impact medical journals: survey of instructions for authors. BMJ. 2005;330:1056–1057
    • View In Article
    • CrossRef
12. Garfield E. The history and meaning of the journal impact factor. JAMA. 2006;295:90–93
    • View In Article
    • CrossRef
13. Lang TA, Secic M. How to Report Statistics in Medicine: Annotated Guidelines for Authors, Editors, and Reviewers. Philadelphia, PA: American Collage of Physicians; 1997;
    • View In Article
14. Altman DG, Gore SM, Gardner MJ, et al. Statistical guidelines for contributors to medical journals. In:  Altman DG,  Machin D,  Bryant TN editor. Statistics with Confidence: Confidence Intervals and Statistical Guidelines. London: BMJ Books; 2000;p. 171–190[Originally published in BMJ 1983. Available online at: http://www.blackwellpublishing.com/content/BPL_Images/Content_store/Sample_chapter/0727913751/0727913751.pdf].
    • View In Article
15. Bailar JC, Mosteller F. Guidelines for statistical reporting in articles for medical journals: amplifications and explanations. Ann Intern Med. 1988;108:266–273
    • View In Article
    • MEDLINE