Abstracts

Article Outline

• Graphics in pharmaceutical advertisements: Are they truthful, are they adequately detailed?
• Effect of written feedback by editors on quality of reviews: Two randomized trials
• Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals
• Author perception of peer review impact of review quality and acceptance on satisfaction
• Content categorization for a peer-reviewed journal

Presented at the Fourth International Congress on Peer Review in Biomedical Publication, Barcelona, Spain, September 2001.

Reprinted with permission from The Journal of the American Medical Association. Reprints not available.

Back to Article Outline

Graphics in pharmaceutical advertisements: Are they truthful, are they adequately detailed? 

Cooper RJ, Schriger DL, Wallace RC, Mikulich VJ, Wilkes MS

(UCLA Emergency Medicine Center [Cooper, Schriger, Wallace, Mikulich] and Department of Medicine [Wilkes], UCLA School of Medicine, Los Angeles, CA)

Objective: Are pharmaceutical advertisements sophisticated medical communications akin to scientific publications, or hollow slogans akin to popular advertising? If they are the former, graphs within advertisements should be similar to graphs in scientific manuscripts. This study characterized the quantity and quality of graphs in pharmaceutical advertisements.

Design: All pharmaceutical advertisements in 10 leading US medical journals published in 1999 were reviewed, and each data graph was evaluated to characterize its features. Pharmaceutical advertisement graphs were contrasted with graphs in articles of The Journal of the American Medical Association (JAMA) and Annals of Emergency Medicine.

Results: There were 836 glossy and 455 small-print pages in 484 unique advertisements (increase of 3,190 total advertisements). Forty-nine percent of glossy page area was nonscientific figures/images, 0.4% tables, and 1.6% scientific graphs (74 graphs in 64 advertisements). The remaining 49% was text or blank page. Eight percent of graphs had errors, 5% had visual obfuscation, and 12% used nonstandard graphing conventions. Only 36% of graphs were self-explanatory. No graphs contained advanced features (pairing, symbolic dimensionality, or small multiples). Fifty-eight percent presented data on an outcome relevant to the drug's indication. When comparing the pharmaceutical advertisement graphs (n=74) with scientific graphs from JAMA (n=64) and Annals of Emergency Medicine (n=128), more simple univariate graphs (96%) were noted in advertisements than in articles published in JAMA (63%) or Annals of Emergency Medicine (53%). Pharmaceutical advertisement graphs had more visual noise (66% versus JAMA 0% or Annals of Emergency Medicine 10%), more numeric distortion (36% versus JAMA 6% or Annals of Emergency Medicine 5%), and more redundancies within the graphs (46% versus JAMA 14% or Annals of Emergency Medicine 16%). The efficiency of data presentation quantified by the data depiction index (area of graph that contains information) was less in the pharmaceutical advertisement graphs; median and interquartile ranges were for pharmaceutical advertisements 0.22 (0.11, 0.43), JAMA 1.1 (0.52, 3.36), and Annals of Emergency Medicine 0.94 (0.54, 1.7).

Conclusions: Graphs in pharmaceutical advertisements were rare and, when present, were of lower quality than those in journal articles. The pharmaceutical advertisement graphs' designs frequently resulted in numeric distortion, which is specifically prohibited by US Food and Drug Administration regulations.

Back to Article Outline

Effect of written feedback by editors on quality of reviews: Two randomized trials 

Callaham ML, Knopp RK, Gallagher EJ (Division of Emergency Medicine, Department of Medicine, University of California, San Francisco, CA[Callaham]; Emergency Department, Regions Hospital, St. Paul, MN [Knopp]; Albert Einstein College of Medicine, Department of Emergency Medicine, Montefiore Medical Center, Bronx, NY [Gallagher]) JAMA. 2002;287:2781-2783

Context: Better peer review is needed, but proven methods to improve quality are unknown. Our objective was to determine whether written feedback to reviewers improves subsequent reviews.

Methods: Eligible reviewers were randomized to intervention or control (receiving other reviewers' unscored reviews and the editor's decision letter). Study 1 (September 1998 to September 2000) included reviewers with a median quality score of 3 or lower; study 2 (April 2000 to January 2002), reviewers with median score of 4 or lower. Study 1 was designed with a power of 0.80 to detect a difference in score of 1; study 2, with a power of 0.80 to detect a difference of 0.5. All reviewers were at a peer-reviewed journal (Annals of Emergency Medicine ). The main outcome measure was the editor's routine quality rating (1 to 5) of all reviews (blinded to study enrollment).

Results: For study 1, 51 reviewers were eligible and randomized and 35 had sufficient data (182 reviews) for analysis. The mean individual reviewer rating change was 0.16 (95% confidence interval [CI] −0.26 to 0.58) for control and −0.13 (95% CI −0.49 to 0.23) for intervention. For study 2, 127 reviewers were eligible and randomized, and 95 had sufficient data (324 reviews). Controls had a mean individual rating change of 0.12 (95% CI −0.20 to 0.26) and intervention reviewers, 0.06 (−0.19 to 0.31).

Conclusions: In study 1, minimal feedback from editors on review quality had no effect on subsequent performance of poor-quality reviewers, and the trend was toward a negative effect. In study 2, feedback to average reviewers was more extensive and supportive but produced no improvement in reviewer performance. Simple written feedback to reviewers seems to be an ineffective educational tool.

Back to Article Outline

Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals 

Callaham M, Wears RL, Weber E (Department of Emergency Medicine, University of California, San Francisco, CA [Callaham, Weber]; Department of Emergency Medicine, University of Florida Health Science Center, Jacksonville FL [Wears]) JAMA. 2002;287:2847-2850

Context: Citation by other authors is important in the dissemination of published science, but factors predicting it are little studied.

Methods: To identify characteristics of published research predicting citation in other journals, we searched the Science Citations Index database for a standardized 3.5 years for all citations of published articles originally submitted to a 1991 emergency medicine specialty meeting. Analysis was conducted by classification and regression trees, a nonparametric modeling technique of regression trees, to determine the impact of previously determined characteristics of the full articles on the outcome measures. We calculated the number of times an article was cited each year and calculated the mean impact factor (citations per manuscript per year) in other citing journals.

Results: Of the 493 submitted manuscripts, 204 published articles met entry criteria. The mean citations per year was 2.04 (95% confidence interval 1.6 to 2.4; range 0 to 20.9) in 440 different journals. Nineteen articles (9.3%) were never cited. The ability to predict the citations per year was weak (pseudo R2=0.14.). The strongest predictor of citations per year was the impact factor of the original publishing journal. The presence of a control group, the subjective newsworthiness score, and sample size predicted citation frequency (24.3%, 26.0%, and 26.5% as strongly, respectively). The ability to predict mean impact factor of the citing journals was even weaker (pseudo R2=0.09). The impact factor of the publishing journal was the strongest predictor, followed by the newsworthiness score (89.9% as strongly) and a subjective quality score (61.5%). Positive outcome bias was not evident for either outcome measure.

Conclusion: In this cohort of published research, commonly used measures of study methodology and design did not predict the frequency of citations or the importance of citing journals. Positive outcome bias was not evident. The impact factor of the original publishing journal was more important than any other variable, suggesting that the journal in which a study is published may be as important as traditional measures of study quality in ensuring dissemination.

Back to Article Outline

Author perception of peer review impact of review quality and acceptance on satisfaction 

Weber EJ, Katz PP, Waeckerle JF, Callaham ML (Division of Emergency Medicine [Weber, Callaham] and Institute for Health Policy Studies [Katz], Department of Medicine, University of California, San Francisco, CA; and Department of Emergency Medicine, Baptist Medical Center, Menorah Medical Center, School of Medicine, University of Missouri, Kansas City, MO [Waeckerle]) JAMA. 2002;287:2790-2793

Context: To determine author perception of peer review and association between quality of review and author satisfaction.

Methods: Survey between May 1999 and October 2000 of 897 corresponding authors of manuscripts under consideration by Annals of Emergency Medicine and who had received final editorial decisions during the study period. A total of 576 authors (64%) returned the survey. Using a 5-point Likert scale, the survey assessed differences in satisfaction between authors whose manuscripts were accepted, reviewed and rejected, and rejected without full review. The association of author satisfaction with editor's assessment of review quality, publication decision, author sex, specialty, and publication experience were also assessed.

Results: Overall mean (SD) satisfaction score, indicated by agreement with “My experience with the review process will make me more likely to submit to Annals in the future,” was 3.1 (1.0) and was significantly higher among authors of accepted papers (3.7 [0.9]) than among either group of rejected papers (rejected/reviewed, 2.8 [1.0]; rejected/no review, 3.0 [0.9]; P <.05). Authors whose manuscripts were reviewed and rejected were the least satisfied with the time to decision (rejected/reviewed, 3.0 [1.2] versus accepted, 3.7 [1.0] and rejected/no review, 3.9 [0.9]; P <.05). Those whose papers were rejected without review were the least satisfied with the letter explaining the editorial decision (rejected/no review, 2.8 [1.2] versus accepted, 4.2 [0.7] and rejected/reviewed, 3.1 [1.2]; P <.05). Among respondents whose manuscripts underwent full review (accepted and rejected/reviewed), overall satisfaction was highly associated with acceptance of the manuscript for publication (odds ratio [OR] 6.12; 95% confidence interval [CI] 3.43 to 10.91) but not with quality rating of reviews (OR 1.26; 95% CI 0.84 to 1.90).

Conclusion: Contributor satisfaction with peer review was modest. Authors of rejected manuscripts were dissatisfied with the time to decision and communication from the editor. Author satisfaction is associated with acceptance but not with review quality.

Back to Article Outline

Content categorization for a peer-reviewed journal 

Newman D, Zou B, Tintinalli J (Department of Emergency Medicine, University of Pittsburgh School of Medicine, Pittsburgh, PA [Newman]; and Department of Biostatistics, School of Public Health [Zou], and Department of Emergency Medicine [Tintinalli], University of North Carolina, Chapel Hill, NC)

Objective: To develop a system for content categorization of manuscripts for a peer-reviewed journal that can also categorize reviewers and editors.

Design: In phase 1, a set of 51 content categories that was developed for reviewer and editor assignments was reorganized by the journal's editorial board, using the Delphi approach. Decision rules for the content categories were defined. The content categories for 4 months of journal publications were then validated. In phase 2, content categories were refined, granular subcategories were developed, and categories were validated for 24 months of publications. The unweighted κ coefficient was determined to measure agreement between 2 reviewers for the content categorization of articles.

Results: In phase 1, 17 content categories were developed with a κ coefficient of 0.79 (95% confidence interval [CI] 0.68 to 0.89; P <.001) for 4 months of publications. In phase 2, content categories were expanded to 26 with an additional 94 subcategories. A total of 248 articles were classified, with 343 observations. Subcategories were necessary to categorize reviewers with focused areas of interest, but were impractical to use for articles. The κ coefficient for agreement between authors was 0.88 (95% CI 0.84 to 0.92; P <.0001) for phase 2. The 26 content categories were then used to assign areas of responsibility for editors.

Conclusions: A single system of content categorization was developed to organize the content of articles, reviewer expertise, and editor responsibility. This method is simple and practical, and while it was developed for a single journal, the principles could be applied to other journals to categorize their unique content.