Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources

      Study objective

      Since 2014, Academic Life in Emergency Medicine (ALiEM) has used the Approved Instructional Resources (AIR) score to critically appraise online content. The primary goals of this study are to determine the interrater reliability (IRR) of the ALiEM AIR rating score and determine its correlation with expert educator gestalt. We also determine the minimum number of educator-raters needed to achieve acceptable reliability.


      Eight educators each rated 83 online educational posts with the ALiEM AIR scale. Items include accuracy, usage of evidence-based medicine, referencing, utility, and the Best Evidence in Emergency Medicine rating score. A generalizability study was conducted to determine IRR and rating variance contributions of facets such as rater, blogs, posts, and topic. A randomized selection of 40 blog posts previously rated through ALiEM AIR was then rated again by a blinded group of expert medical educators according to their gestalt. Their gestalt impression was subsequently correlated with the ALiEM AIR score.


      The IRR for the ALiEM AIR rating scale was 0.81 during the 6-month pilot period. Decision studies showed that at least 9 raters were required to achieve this reliability. Spearman correlations between mean AIR score and the mean expert gestalt ratings were 0.40 for recommendation for learners and 0.35 for their colleagues.


      The ALiEM AIR scale is a moderately to highly reliable, 5-question tool when used by medical educators for rating online resources. The score displays a fair correlation with expert educator gestalt in regard to the quality of the resources. The score displays a fair correlation with educator gestalt.
      To read this article in full you will need to make a payment
      ACEP Member Login
      ACEP Members, full access to the journal is a member benefit. Use your society credentials to access all journal content and features.
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'

      Purchase one-time access:

      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Lang E.S.
        • Wyer P.C.
        • Haynes R.B.
        Knowledge translation: closing the evidence-to-practice gap.
        Ann Emerg Med. 2007; 49: 355-363
        • Arbesman S.
        The Half-Life of Facts: Why Everything We Know Has an Expiration Date.
        Penguin Group, New York, NY2012
        • Christensen C.
        • Raynor M.E.
        The Innovator’s Solution: Creating and Sustaining Successful Growth.
        Harvard Business School Press, Boston, MA2003
        • Paterson Q.S.
        • Thoma B.
        • Milne W.K.
        • et al.
        A systematic review and qualitative analysis to determine quality indicators for health professions education blogs and podcasts.
        J Grad Med Educ. 2015; 7: 549-554
        • Cadogan M.
        • Thoma B.
        • Chan T.M.
        • et al.
        Free open access meducation (FOAM): the rise of emergency medicine and critical care blogs and podcasts (2002-2013).
        Emerg Med J. 2014; 31: e76-e77
        • Thoma B.
        • Chan T.
        • Benitez J.
        • et al.
        Educational scholarship in the digital age: a scoping review and analysis of scholarly products.
        Winnower. 2014; 1: 1-13
        • Purdy E.
        • Thoma B.
        • Bednarczyk J.
        • et al.
        The use of free online educational resources by Canadian emergency medicine residents and program directors.
        CJEM. 2015; 1717: 101-106
        • Mallin M.
        • Schlein S.
        • Doctor S.
        • et al.
        A survey of the current utilization of asynchronous education among emergency medicine residents in the United States.
        Acad Med J Assoc Am Med Coll. 2014; 89: 598-601
        • Pearson D.
        • Bond M.
        • Kegg J.
        • et al.
        Evaluation of social media use by emergency medicine residents and faculty.
        West J Emerg Med. 2015; 16: 715-720
      1. Thurtle N, Banks C, Cox M, et al. Free open access medical education resource knowledge and utilisation amongst emergency medicine trainees: a survey in four countries. African J Emerg Med.

        • Scott K.R.
        • Hsu C.H.
        • Johnson N.J.
        • et al.
        Integration of social media in emergency medicine residency curriculum.
        Ann Emerg Med. 2014; 64: 396-404
        • Hayes B.D.
        • Kobner S.
        • Trueger N.S.
        • et al.
        Social media in the emergency medicine residency curriculum: social media responses to the Residents’ Perspective article.
        Ann Emerg Med. 2015; 65: 573-583
      2. Cadogan M, Nickson C. Free open access medical education. Life in the Fast Lane. Available at: Published December 4, 2014. Accessed January 9, 2016.

        • Weingart S.D.
        • Thoma B.
        The online hierarchy of needs: a beginner’s guide to medical social media and FOAM.
        Emerg Med Australas. 2015; 27: 5
        • Thoma B.
        • Chan T.
        • Desouza N.
        • et al.
        Implementing peer review at an emergency medicine blog: bridging the gap between educators and clinical experts.
        CJEM. 2015; 17: 188-191
        • Thoma B.
        • Sanders J.L.
        • Lin M.
        • et al.
        The Social Media Index: measuring the impact of emergency medicine and critical care websites.
        West J Emerg Med. 2015; 16: 242-249
        • Charnock D.
        • Shepperd S.
        • Needham G.
        • et al.
        DISCERN: an instrument for judging the quality of written consumer health information on treatment choices.
        J Epidemiol Community Health. 1999; 53: 105-111
      3. Lin M, Joshi N, Grock A, et al. Approved Instructional Resources (AIR) series: a national initiative to identify quality emergency medicine blog and podcast content for resident education. J Grad Med Educ.

        • Worster A.
        • Kulasegaram K.
        • Carpenter C.R.
        • et al.
        Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.
        Acad Emerg Med. 2011; 18: 1193-1200
        • Carpenter C.R.
        • Sarli C.C.
        • Fowler S.A.
        • et al.
        Best Evidence in Emergency Medicine (BEEM) rater scores correlate with publications’ future citations.
        Acad Emerg Med. 2013; 20: 1004-1012
        • Webb N.M.
        • Shavelson R.J.
        Generalizability theory: overview.
        in: Wiley StatsRef: Statistics Reference Online. John Wiley & Sons, Ltd, Hoboken, NJ2014
        • Downing S.M.
        Validity: on the meaningful interpretation of assessment data.
        Med Educ. 2003; 37: 830-837
        • Schuwirth L.W.T.
        • van der Vleuten C.P.M.
        Programmatic assessment and Kane’s validity perspective.
        Med Educ. 2012; 46: 38-48
        • Linzer M.
        The journal club and medical education: over one hundred years of unrecorded history.
        Postgrad Med J. 1987; 63: 475-478
        • Thangasamy I.A.
        • Leveridge M.
        • Davies B.J.
        • et al.
        International Urology Journal Club via Twitter: 12-month experience.
        Eur Urol. 2014; 66: 112-117
        • Choo E.K.
        • Ranney M.L.
        • Chan T.M.
        • et al.
        Twitter as a tool for communication and knowledge exchange in academic medicine: a guide for skeptics and novices.
        Med Teach. 2015; 37: 411-416
        • Melvin L.
        • Chan T.M.
        Using Twitter in clinical education and practice.
        J Grad Med Educ. 2014; 6: 581-582
        • Chan T.M.
        • Rosenberg H.
        • Lin M.
        Global Emergency Medicine Journal Club: social media responses to the January 2014 online emergency medicine journal club on subarachnoid hemorrhage.
        Ann Emerg Med. 2014; 64: 88-94
        • Thoma B.
        • Rolston D.
        • Lin M.
        Global Emergency Medicine Journal Club: social media responses to the March 2014 Annals of Emergency Medicine journal club on targeted temperature management.
        Ann Emerg Med. 2014; 64: 207-212
        • Radecki R.P.
        • Rezaie S.R.
        • Lin M.
        Global Emergency Medicine Journal Club: social media responses to the November 2013 Annals of Emergency Medicine journal club.
        Ann Emerg Med. 2014; 63: 490-494
        • Rezaie S.R.
        • Swaminathan A.
        • Chan T.
        • et al.
        Global Emergency Medicine Journal Club: a social media discussion about the Age-Adjusted D-Dimer Cutoff Levels to Rule Out Pulmonary Embolism trial.
        Ann Emerg Med. 2015; 65: 604-613
        • Westafer L.
        • Hensley J.
        • Shaikh S.
        • et al.
        Global Emergency Medicine Journal Club: a social media discussion about the lack of association between Press Ganey scores and emergency department analgesia.
        Ann Emerg Med. 2015; 66: 1-7
        • Chan T.M.
        • Thoma B.
        • Radecki R.
        • et al.
        Ten steps for setting up an online journal club.
        J Contin Educ Health Prof. 2015; 35: 148-154
        • Chan T.
        • Joshi N.
        • Lin M.
        • et al.
        Using Google Hangouts on Air for medical education: a disruptive way to leverage and facilitate remote communication and collaboration.
        J Grad Med Educ. 2015; 7: 171-173
        • Lin M.
        • Thoma B.
        • Trueger N.S.
        • et al.
        Quality indicators for blogs and podcasts used in medical education: modified Delphi consensus recommendations by an international cohort of health professions educators.
        Postgrad Med J. 2015; 91: 546-550
        • Thoma B.
        • Chan T.M.
        • Paterson Q.S.
        • et al.
        Emergency medicine and critical care blogs and podcasts: establishing an international consensus on quality.
        Ann Emerg Med. 2016; 66: 396-402
        • Guyatt G.
        • Meade M.O.
        • Rennie D.
        • et al.
        Users’ Guides to the Medical Literature: A Manual for Evidence-Based Clinical Practice.
        3rd ed. McGraw-Hill Education, 2015