The Cochrane Library as a Resource for Evidence on Out-of-Hospital Health Care Interventions

Article Outline

• Abstract
• Introduction
  • Background
  • Importance
  • Goals of This Investigation
• Materials and Methods
  • Study Design
  • Primary Data Analysis
• Results
• Limitations
• Discussion
• Acknowledgment
• References
• Copyright

Study objective

Although the Cochrane Library is promoted as a rigorous source of evidence, the relevance of this evidence to the out-of-hospital setting has not been assessed. The objective of this study is to identify existing controlled trials and systematic reviews in the Cochrane Library that have been conducted in or are relevant to out-of-hospital health care. The scope of out-of-hospital care, the years and trends of out-of-hospital research publication, and the journals of publication are also examined.

Methods

Using a search strategy developed by the Cochrane Prehospital and Emergency Health Field, 2 reviewers searched issue 4, 2005 of the Cochrane Library to identify reports of controlled trials and systematic reviews on out-of-hospital interventions. Three independent reviewers screened the titles identified by the search strategy and applied predetermined criteria to classify the reported study as out-of-hospital based or not out-of-hospital based. The out-of-hospital-based studies were then categorized as randomized or nonrandomized trials.

Results

Screening of the 19,759 titles retrieved by the search strategy identified 4,016 studies that were potentially out-of-hospital based. Abstract and full-text analysis of the 4,016 studies identified 400 reports of trials and 13 reviews or protocols that were out-of-hospital based. Of the 400 reports of trials identified, 299 (75%) were randomized trials. The number of out-of-hospital trials published increased steadily from the 1970s to the late 1990s, with the majority of trials (63%) covering interventions related to resuscitation and cardiac care. Annals of Emergency Medicine published more out-of-hospital trials than any other journal, followed by Resuscitation and Journal of the American Medical Association.

Conclusion

The Cochrane Library provides a useful resource of health care evidence; however, the relatively small number of out-of-hospital-based systematic reviews and trials does not comprehensively cover the broad scope of out-of-hospital health care.

SEE EDITORIAL, P. 351.

Back to Article Outline

Introduction 

Background 

During the last 10 years, the focus of out-of-hospital care has shifted dramatically from simple treatment and transport roles to complex and sophisticated clinical interventions. This shift in focus has led to an increased need to base policy, practice, and delivery decisions on the highest possible levels of evidence.

A prerequisite to adopting any evidence-based approach in health care is the need to assemble a body of evidence derived from the results of rigorous studies. This body of evidence, in the form of controlled trials and systematic reviews, should be easily accessible to facilitate implementation into education, practice, and delivery.

Importance 

The Cochrane Library, the primary output of the Cochrane Collaboration, is a potential source of this evidence. However, identifying evidence for out-of-hospital interventions in the Cochrane Library can be challenging. Medical Subject Headings (MeSH) are lacking for out-of-hospital interventions, and conducting a comprehensive search with text terms can be laborious because the scope of out-of-hospital practice crosses many medical disciplines,¹ requiring the use of a large number of search terms to identify all existing literature.

In 2004, the Cochrane Prehospital and Emergency Health Field was registered as an official entity of the Cochrane Collaboration. A Cochrane field is an entity that focuses on a dimension of health care rather than on a specific health care problem. One of the functions of the Cochrane Prehospital and Emergency Health Field is to develop and maintain a register of studies relevant to the areas of out-of-hospital and emergency health care. In the process of developing this register a search strategy was designed by members of the field's advisory board to identify systematic reviews and reports of trials in the Cochrane Library that were based on research that had been conducted in the out-of-hospital environment.²

To help prioritize searching and to avoid the duplication of previous studies, the authors adapted the search strategies used by previous research groups who had conducted literature searches for out-of-hospital research in databases other than the Cochrane Library.³, ⁴, ⁵, ⁶, ⁷ This strategy involved adapting search terms from other databases (MEDLINE and CINAHL) for the Cochrane Library (MeSH are not consistent across databases) and adding search terms that had not been used before.

Goals of This Investigation 

The aim of this study was to use the search strategy developed by the field to identify reports of randomized controlled trials, nonrandomized controlled trials, and systematic reviews in the Cochrane Library that were conducted in the out-of-hospital environment. The scope of out-of-hospital care, the years and trends of publication of out-of-hospital research, and the journals of publication are also examined.

Back to Article Outline

Materials and Methods 

Study Design 

The methodology used for this comprehensive literature review is summarized in Figure 1.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 1. 

  Study methodology and selection of included research publications.

We conducted a search of the Cochrane Library, issue 4, 2005, using a search strategy designed specifically for the Cochrane Library.² The Cochrane Library contains a register of reports of controlled trials called the Cochrane Central Register of Controlled Trials, which is developed through regular, comprehensive database searches of EMBASE, MEDLINE, and CINAHL, supplemented by hand-searching efforts in the Cochrane contributors. In addition to Cochrane Central Register of Controlled Trials, the Cochrane Library contains 2 databases of systematic reviews: the Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effect (DARE). Each of these databases was searched by using the same search strategy.

The search strategy was tested in the Cochrane Library with a sample of 84 randomized controlled trials of out-of-hospital health care interventions that had been identified from a previous literature search.⁷ The original search strategy identified 83 of the sample trials, resulting in a further search term being added to the strategy (“out of hospital” needed to be listed in the search strategy with and without hyphens because each specific way identified trials that the other did not). When this further term was added, the search strategy identified all 84 sample trials. The search strategy is summarized in Figure 2.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 2. 

  Description of search strategy used to search the Cochrane Library for out-of-hospital reviews and trials.

For the purpose of this study, we defined a study as “out-of-hospital based” if it was conducted in the out-of-hospital environment and involved out-of-hospital patients or out-of-hospital professionals up to the point of admission to the hospital emergency department (ED).

Titles of records retrieved by the search were classified with a 3-category scale, as described in Figure 3. Further keywords (“prehospital,” “pre-hospital,” “out-of-hospital,” “out of hospital,” “ambulance,” “paramedic,” “EMS”) were used to assist in the identification of out-of-hospital-based studies during the initial search. Because the search strategy designed for this review was deliberately sensitive, the majority of trials and systematic reviews identified were clearly not out-of-hospital based (did not have any of the key words throughout the title, abstract, or text, and methodology clearly indicated that the trial or review had not been conducted in the out-of-hospital environment or included out-of-hospital patients). The remaining trials were either clearly out-of-hospital (used key words in title or abstract) or were “unclear” until methodology had been reviewed.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 3. 

  Three-category scale used to select the included studies.

The trials identified by the search strategy were further classified according to whether the report was of a randomized or a nonrandomized controlled trial. Any trial that reported randomization, irrespective of the methodology, was categorized as randomized. Any trial that did not report randomization but contained a prospectively collected comparison group was categorized as a nonrandomized controlled trial. Two independent reviewers (E.S. and T.O.) classified the studies as a randomized or a nonrandomized controlled trial by reviewing abstracts of all reports of trials and full text of studies in which the study methodology could not be determined from the abstract alone. Because the Cochrane Library includes only reports on controlled trials, no retrospective or low-level evidence was searched for or included in this review.

Titles of the systematic reviews (including protocols for systematic reviews) and reports of trials identified by the search were independently assessed by 3 reviewers (E.S., P.J., and C.M.). The full text of all systematic reviews identified by the search was assessed for inclusion. Abstracts of trials that appeared to have been conducted in the out-of-hospital environment were independently assessed by 2 reviewers (E.S. and P.J.). Duplicate records or records commenting on previously published studies were removed so that only original reports of trials were included. Where required, the full-text article for the trial was obtained. Disagreement was resolved through discussion and consultation among the reviewers.

Primary Data Analysis 

Agreement was measured and reported as simple agreement. Consensus process was completed, including an additional adjudicator when needed, until agreement was 100%. Descriptive statistics are presented with counts reported as percentages.

Because this study used an existing, publicly accessible database, no ethics approval was sought for this project. No authors were contacted during this study.

Back to Article Outline

Results 

As of issue 4, 2005, the Cochrane Library contained 463,763 Cochrane Central Register of Controlled Trials records, 4,113 systematic reviews in CDSR, and 5,585 reviews in the DARE database.

The initial search identified a total of 19,759 potentially relevant records in the Cochrane Library. A title and abstract search of these records identified 4,016 that were potentially out-of-hospital-based reviews or trials. The title and abstract search was conducted by 3 reviewers (E.S., P.J., and C.M.), who had extensive knowledge of out-of-hospital health care interventions. If the title or abstract did not mention any of the search terms or an intervention that was used by out-of-hospital health care professionals, or if it was clear that the study was not out-of-hospital research, then the study was excluded.

Protocols and systematic reviews in CDSR accounted for 188 of the 4,016 records, and 3,823 were reports of trials in the Cochrane Central Register of Controlled Trials database. Abstract and full-text search of the 4,016 potentially out-of-hospital-based studies identified a total of 413 studies conducted in the out-of-hospital environment.

Of the 413 out-of-hospital reports, 13 were systematic reviews, which equates to less than 1% of the total number of systematic reviews in the Cochrane Library (13/4,113). Eight of these were Cochrane systematic reviews (3 protocols and 5 complete reviews) of out-of-hospital health care interventions, and 5 were abstracts of other out-of-hospital-based systematic reviews from the DARE database. The 8 Cochrane systematic reviews (including protocols) were published by the Cochrane Anaesthesia, Heart and Injuries Groups and are listed in Table 1.

Table 1. Review group of publication for the Cochrane Reviews available in the Cochrane Library.

Initial agreement between the 3 reviewers when out-of-hospital-based protocols and systematic reviews were identified was 85%, with discussion among the reviewers leading to 100% agreement. Initial agreement between the 3 reviewers for reports of out-of-hospital-based trials was 68%, with discussion among the 3 reviewers and consultation with a fourth reviewer leading to 100% agreement. A fourth reviewer was consulted on 23 (0.6%) of the 4,016 potential out-of-hospital-based trials identified by the search.

Of the 413 out-of-hospital reports, 400 were reports of trials in the Cochrane Central Register of Controlled Trials database, which equates to 0.1% of all controlled trials reported in the Cochrane Library (400/463,763). Highlighting the broad scope of out-of-hospital health care, the reports of trials identified in Cochrane Central Register of Controlled Trials covered many clinical areas. For the purpose of this paper, we separated the reports of trials identified into the following clinical subgroups: resuscitation, cardiac, trauma, respiratory, pediatric, psychiatric care, other medical, service delivery, training, and education. Of the trials identified, 144 (36%) related to resuscitation, 96 (24%) related to cardiac care, and 45 (11%) related to trauma care. Of the 400 reports of trials retrieved from the Cochrane Central Register of Controlled Trials, 299 (75%) were randomized controlled trials and 101 (25%) were nonrandomized controlled trials.

The reports of out-of-hospital-based trials were then separated into year of publication (Figure 4). Publication of out-of-hospital-based trials increased steadily from 1980, peaking in 1999 with 36 publications. Despite 33 publications in 2003, however, the number of out-of-hospital trials published since 2000 continues to decrease.

• 
  • View Large Image
  • Download to PowerPoint

• Figure 4. 

  Number of out-of-hospital-based trials by year of publication.

The trials were then reviewed (E.S. and T.O.) for journal of publication. Highlighting the broad scope of the out-of-hospital field, out-of-hospital-based trials were published in 120 journals. Table 2 indicates those journals that published the greatest number of out-of-hospital-based trials. The majority (48%) of the out-of-hospital-based trials retrieved were conducted in the United States, followed by France (8%), Germany (7.9%), and the United Kingdom (6.7%). Of the 400 reports of trials retrieved, 365 (91%) were in the English language, and 35 (9%) were in languages other than English. Nine of the trials retrieved were conference reports, with the remaining 391 trials being journal articles.

Table 2. Leading journals for publication of out-of-hospital trials identified in this review.

Back to Article Outline

Limitations 

The results of our study are subject to some limitations. First, the results of the search strategy were analyzed by only 3 reviewers, and the methodology used to classify a study as out-of-hospital based could be open to debate. We are confident, however, that the definitions and inclusion-exclusion criteria provided to the reviewers mean that we have consistently identified out-of-hospital-based research as defined for the purposes of this study.

Second, the quality of the included systematic reviews and trials was not evaluated, and we recognize the need for further research in this area.

Third, international differences in definitions and terminology for “out-of-hospital” may introduce a limitation to this study. We are confident, however, that the sensitive search strategy that we used for this study would have been sufficient to identify all relevant studies. One problem with designing such a sensitive search strategy, however, is the consequent lack of specificity. The search results included many irrelevant records, which decreases the everyday “usability” of the search strategy. For the purpose of this study, the reviewers used an optimally sensitive search strategy at the expense of specificity to identify as many out-of-hospital studies as possible.

Finally, these results are subject to changes as new research is published. The original search was updated, and we feel confident that the majority of the evidence is included in this report.

Back to Article Outline

Discussion 

Past research has determined that systematic reviews in the Cochrane Library had relevance to emergency health care practitioners¹; however, this research used a very early version of the Library (<1,000 reviews), and the authors did not focus on out-of-hospital care. The current study searched the current Cochrane Library (>3,000 reviews) and focused exclusively on out-of-hospital evidence in the form of out-of-hospital-based trials and systematic reviews. To our knowledge, this is the first study to use a highly sensitive search strategy to comprehensively search the contents of the Cochrane Library for out-of-hospital-based research.

The principal finding of this study is the contrast between the wide scope of the out-of-hospital field (resuscitation, airway diseases, injury, out-of-hospital medical treatments, etc) and the lack of high-quality evidence on which to guide practice. Nearly two thirds (63%) of the 400 reports of trials relevant to out-of-hospital care identified in the Cochrane Library were on interventions concerning resuscitation and cardiac care. Although taking nothing away from the quality of the research in this area, cardiac arrests and acute resuscitative attempts account for only 2% of all ambulance responses in most emergency medical service systems.¹⁶ Therefore, the majority of health care interventions used in the out-of-hospital environment are not based on strong evidence from controlled studies. Further evidence of this is the paucity of out-of-hospital trials included in systematic reviews. The 5 completed out-of-hospital-based Cochrane systematic reviews identified in this study contain a small number of out-of-hospital trials on which to base conclusions.

Another issue identified by this study is the apparent decrease in the number of out-of-hospital trials published since 2000. This finding needs to be interpreted cautiously because there can be impressive delays from the publication of a trial in a journal (especially if the journal is not indexed in MEDLINE or EMBASE) and its subsequent inclusion in Cochrane Central Register of Controlled Trials. In defining “out-of-hospital” for the purpose of this study, we included any study that was conducted in the “prehospital” or “out-of-hospital” field, which involved out-of-hospital patients or practitioners, up to the point of admission to the hospital ED. Therefore, any study that had been conducted in an ED setting, regardless of whether it was investigating a health care intervention relevant to out-of-hospital care, was excluded in the results of this study.

To our knowledge, this is the first time that the evidence in the Cochrane Library has been assessed for relevance to out-of-hospital health care interventions. Although the number of systematic reviews and reports of out-of-hospital trials identified is low in comparison to that of other medical specialities, the number of trials identified by this study is much larger than those reported in previous out-of-hospital literature searching efforts.³, ⁴, ⁵, ⁷, ⁸, ¹⁷, ¹⁸

In conclusion, although the Cochrane Library is a valuable resource for out-of-hospital interventions, the topic coverage is sparse. This study highlights that the Cochrane Library provides 3 useful databases (Cochrane Central Register of Controlled Trials, CDSR, DARE) for out-of-hospital practitioners, researchers, educators, and managers to use to identify the current evidence base and the gaps for future original research and systematic reviews. The study also identifies the journals that publish the majority of out-of-hospital-based randomized and nonrandomized controlled trials. This information could assist out-of-hospital professionals to identify priority journals among the overwhelming amount of literature that is published annually.

Out-of-hospital providers would likely find this resource even more useful if more of the key topic areas were covered. Increased involvement in the Cochrane Collaboration should help to increase the number of reviews conducted on out-of-hospital-relevant questions, enhancing the value of this resource in facilitating evidence-based out-of-hospital care and teaching. The efforts of the Cochrane Prehospital and Emergency Health Field will now focus on facilitating the preparation of systematic reviews of the out-of-hospital trials identified by this study as a way of strengthening the evidence base for the use of out-of-hospital interventions.

Back to Article Outline

We acknowledge the assistance of Brian Rowe, MD, Matthew Cooke, PhD, Michael Sayre, MD, and Jason Wasiak, MPH, in the preparation of this search strategy and article.⁹, ¹⁰, ¹¹, ¹², ¹³, ¹⁴

Back to Article Outline

References 

1. Emond SD, Wyer PC, Brown MD, et al. How relevant are the systematic reviews in the Cochrane Library to emergency medical practice?. Ann Emerg Med. 2002;39:153–158
   • View In Article
   • Abstract
   • Full Text
   • Full-Text PDF (77 KB)
   • CrossRef
2. Smith E, McDonald S, Wasiak J, et al. The development of a prehospital search filter for the Cochrane Library [serial on the Internet; 2004;2:CC990071]. Available at: http://www.jephc.com/full_article.cfm?content_id=152. Accessed January 2, 2006.
   • View In Article
3. Langham J, Thompson E, Rowan K. Identification of randomized controlled trials from the emergency medicine literature: comparison of handsearching versus MEDLINE searching. Ann Emerg Med. 1999;34:25–34
   • View In Article
   • Abstract
   • Full Text
   • Full-Text PDF (82 KB)
   • CrossRef
4. Brazier H, Murphy AW, Lynch C, et al. Searching for the evidence in pre-hospital care: a review of randomized controlled trials. J Accid Emerg Med. 1999;16:18–23
   • View In Article
   • MEDLINE
5. Bunn F, Kwan I, Roberts I, et al. Effectiveness of Pre-Hospital Trauma Care: Report to the World Health Organization Pre-Hospital Steering Committee. Geneva, Switzerland: World Health Organization; 2001.
   • View In Article
6. Kwan I, Bunn F, Roberts I, et al. The Development of a Register of Randomized Controlled Trials in Pre-hospital Trauma Care. World Health Organization, Pre-hospital Steering Committee; 2002.
   • View In Article
7. Sayre M, Brown L, White L. National EMS research agenda: the National Highway Traffic Safety Administration, December 2001. Available at: http://www.researchagenda.org/Agenda/Index.htm (Appendix F). Accessed.
   • View In Article
8. Jacobs I, Finn J. Adrenaline and vasopressin for cardiac arrest [protocol]. Cochrane Database Syst Rev. 2001; Issue 3.
   • View In Article
9. Smith E, Wasiak J, Boyle M. Prophylactic antiemetic therapy in the emergency and ambulance setting for preventing opioid induced nausea and vomiting [protocol]. Cochrane Database Syst Rev. 2004; issue 3.
   • View In Article
10. Lunxian T, Hu X, Qing H. Intravenous amiodarone for treatment of ventricular tachycardia and ventricular fibrillation [protocol]. Cochrane Database Syst Rev. 2003; issue 2.
    • View In Article
11. Lafuente-Lafuente C, Melero-Bascones M. Active chest compression-decompression for cardiopulmonary resuscitation. Cochrane Database Syst Rev. 2004; issue 4.
    • View In Article
12. Sethi D, Kwan I, Kelly AM, et al, for the WHO Pre-Hospital Trauma Care Steering Committee. Advanced trauma life support training for ambulance crews. Cochrane Database Syst Rev. 2001; issue 2.
    • View In Article
13. Bunn F, Roberts I, Tasker R. Hypertonic versus near isotonic crystalloid for fluid resuscitation in critically ill patients. Cochrane Database Syst Rev. 2004; issue 3.
    • View In Article
14. Kwan I, Bunn F, Roberts I, for the WHO Pre-Hospital Trauma Care Steering Committee. Spinal immobilisation for trauma patients. Cochrane Database Syst Rev. 2001; issue 2.
    • View In Article
15. Kwan I, Bunn F, Roberts I, for the WHO Pre-Hospital Trauma Care Steering Committee. Timing and volume of fluid administration for patients with bleeding. Cochrane Database Syst Rev. 2003; issue 3.
    • View In Article
16. McLean SA, Maio RF, Spaite DW, et al. Emergency medical services outcomes research: evaluating the effectiveness of prehospital care. Prehosp Emerg Care. 2002;6:S52–S56
    • View In Article
    • MEDLINE
    • CrossRef
17. Wilson S, Cooke M, Morrell R, et al. A systematic review of the evidence supporting the use of priority dispatch of emergency ambulances. Prehosp Emerg Care. 2002;6:42–49
    • View In Article
    • MEDLINE
    • CrossRef
18. Brice JH, Garrison HG, Evans AT. Study design and outcomes in out-of-hospital emergency medicine research: a ten year analysis. Prehosp Emerg Care. 2000;4:144–150
    • View In Article
    • MEDLINE
    • CrossRef