Advertisement

Lung Ultrasonography for the Diagnosis of SARS-CoV-2 Pneumonia in the Emergency Department

      Study objective

      Accurate diagnostic testing to identify severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is critical. Although highly specific, SARS-CoV-2 reverse transcriptase–polymerase chain reaction (RT-PCR) has been shown in clinical practice to be affected by a noninsignificant proportion of false-negative results. This study seeks to explore whether the integration of lung ultrasonography with clinical evaluation is associated with increased sensitivity for the diagnosis of coronavirus disease 2019 pneumonia, and therefore may facilitate the identification of false-negative SARS-CoV-2 RT-PCR results.

      Methods

      This prospective cohort study enrolled consecutive adult patients with symptoms potentially related to SARS-CoV-2 infection who were admitted to the emergency department (ED) of an Italian academic hospital. Immediately after the initial assessment, a lung ultrasonographic evaluation was performed and the likelihood of SARS-CoV-2 infection, based on both clinical and lung ultrasonographic findings (“integrated” assessment), was recorded. RT-PCR SARS-CoV-2 detection was subsequently performed.

      Results

      We enrolled 228 patients; 107 (46.9%) had SARS-CoV-2 infection. Sensitivity and negative predictive value of the clinical–lung ultrasonographic integrated assessment were higher than first RT-PCR result (94.4% [95% confidence interval {CI} 88.2% to 97.9%] versus 80.4% [95% CI 71.6% to 87.4%] and 95% [95% CI 89.5% to 98.2%] versus 85.2% [95% CI 78.3% to 90.6%], respectively). Among the 142 patients who initially had negative RT-PCR results, 21 tested positive at a subsequent molecular test performed within 72 hours. All these false-negative cases were correctly identified by the integrated assessment.

      Conclusion

      This study suggests that, in patients presenting to the ED with symptoms commonly associated with SARS-CoV-2 infection, the integration of lung ultrasonography with clinical evaluation has high sensitivity and specificity for coronavirus disease 2019 pneumonia and it may help to identify false-negative results occurring with RT-PCR.
      What is already known on this topic
      Reverse transcriptase–polymerase chain reaction testing has high specificity but imperfect sensitivity.
      What question this study addressed
      Does lung ultrasonography combined with initial clinical examination increase the sensitivity for diagnosing coronavirus disease 2019?
      What this study adds to our knowledge
      In this prospective study of 228 patients (47% positive rate), lung ultrasonography combined with clinical examination had a higher sensitivity than the initial reverse transcriptase–polymerase chain reaction test (94% versus 80%).
      How this is relevant to clinical practice
      Lung ultrasonography combined with clinical examination may help identify cases missed on initial reverse transcriptase–polymerase chain reaction testing.

      Introduction

       Background

      Accurate diagnostic testing to identify individuals presenting with symptoms associated with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection is critical not only for appropriate patient management but also for effective and judicious implementation of infection prevention and control strategies.
      • West C.P.
      • Montori V.M.
      • Sampathkumar P.
      COVID-19 testing: the threat of false-negative results.
      • Sharfstein J.M.
      • Becker S.J.
      • Mello M.M.
      Diagnostic testing for the novel coronavirus.
      • Cheng M.P.
      • Papenburg J.
      • Desjardins M.
      • et al.
      Diagnostic testing for severe acute respiratory syndrome–related coronavirus 2: a narrative review.
      A test unable to accurately identify individuals affected by coronavirus disease 2019 (COVID-19) (ie, false-negative result) would generate a significant risk of disease transmission and further propagation of the infection in the community and health care settings,
      • West C.P.
      • Montori V.M.
      • Sampathkumar P.
      COVID-19 testing: the threat of false-negative results.
      ,
      • Marcotte L.M.
      • Liao J.M.
      Incorporating test characteristics into SARS-CoV-2 testing policy—sense and sensitivity.
      ,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      As SARS-CoV-2 tests become more available, understanding how they perform in actual settings (eg, busy emergency departments [EDs]) and developing integrated diagnostic pathways able to identify or minimize false-negative results represent a priority as we prepare for a second wave of infection.
      • West C.P.
      • Montori V.M.
      • Sampathkumar P.
      COVID-19 testing: the threat of false-negative results.
      ,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.

       Importance

      Currently, the most commonly used diagnostic test for the diagnosis of SARS-CoV-2 infection is reverse transcriptase–polymerase chain reaction (RT-PCR) performed with nasopharyngeal swabs.
      • Cheng M.P.
      • Papenburg J.
      • Desjardins M.
      • et al.
      Diagnostic testing for severe acute respiratory syndrome–related coronavirus 2: a narrative review.
      ,
      COVID-19: EU recommendations for testing strategies.
      European Centre for Disease Prevention and Control
      An overview of the rapid test situation for COVID-19 diagnosis in the EU/EEA: technical report, April 1st, 2020.
      • Sethuraman N.
      • Jeremiah S.S.
      • Ryo A.
      Interpreting diagnostic tests for SARS-CoV-2.
      Although highly specific, this test has been shown in clinical practice to be affected by a noninsignificant proportion of false-negative results,
      • West C.P.
      • Montori V.M.
      • Sampathkumar P.
      COVID-19 testing: the threat of false-negative results.
      ,
      • Winichakoon P.
      • Chaiwarith R.
      • Liwsrisakun C.
      • et al.
      Negative nasopharyngeal and oropharyngeal swabs do not rule out COVID-19.
      potentially related to insufficient viral copies, deficient sampling, inappropriate timing in relation to illness onset, or different location of viral replication (eg, lower respiratory tract).
      • Winichakoon P.
      • Chaiwarith R.
      • Liwsrisakun C.
      • et al.
      Negative nasopharyngeal and oropharyngeal swabs do not rule out COVID-19.
      ,
      • Xie X.
      • Zhong Z.
      • Zhao W.
      • et al.
      Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing.
      Considering the unsatisfactory results obtained with RT-PCR,
      • Cheng M.P.
      • Papenburg J.
      • Desjardins M.
      • et al.
      Diagnostic testing for severe acute respiratory syndrome–related coronavirus 2: a narrative review.
      ,
      • Winichakoon P.
      • Chaiwarith R.
      • Liwsrisakun C.
      • et al.
      Negative nasopharyngeal and oropharyngeal swabs do not rule out COVID-19.
      many experts recommend continuing to isolate patients with a high clinical suspicion for disease,
      Centers for Disease Control and Prevention
      Algorithm to assist in the interpretation of influenza testing results and clinical decision-making during periods when influenza viruses are circulating in the community.
      Centers for Disease Control and Prevention
      Interim infection prevention and control recommendations for patients with suspected or confirmed coronavirus disease 2019 (COVID-19) in healthcare settings.
      Centers for Disease Control and Prevention
      Discontinuation of transmission-based precautions and disposition of patients with COVID-19 in healthcare settings (interim guidance).
      an intervention often problematic, given limitations in appropriate isolation spaces and shortages in personal protective equipment; and also recommend conducting further testing, which can be either time consuming (eg, serial RT-PCR tests) or logistically challenging (eg, bronchoscopy for bronchoalveolar lavage, chest computed tomographic [CT]) scan
      • Xie X.
      • Zhong Z.
      • Zhao W.
      • et al.
      Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing.
      ).
      Lung ultrasonography is a noninvasive bedside tool that has been shown to be useful for the assessment of patients presenting with acute respiratory failure,
      • Lichtenstein D.
      • MéZièRe G.
      • Biderman P.
      • et al.
      The comet-tail artifact: an ultrasound sign of alveolar-interstitial syndrome.
      as well as for the diagnosis of several pleural and pulmonary diseases, including acute heart failure, pleural effusion, pneumonia, and pneumothorax.
      • Pivetta E.
      • Goffi A.
      • Nazerian P.
      • et al.
      Lung ultrasound integrated with clinical assessment for the diagnosis of acute decompensated heart failure in the emergency department: a randomized controlled trial.
      • Volpicelli G.
      • Elbarbary M.
      • Blaivas M.
      • et al.
      International evidence-based recommendations for point-of-care lung ultrasound.
      • Maw A.M.
      • Hassanin A.
      • Ho P.M.
      • et al.
      Diagnostic accuracy of point-of-care lung ultrasonography and chest radiography in adults with symptoms suggestive of acute decompensated heart failure: a systematic review and meta-analysis.
      Preliminary reports have been published on the use of lung ultrasonography in the evaluation of patients with SARS-CoV-2 infection.
      • Peng Q.-Y.
      • Wang X.-T.
      • Zhang L.-N.
      Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.
      • Xing C.
      • Li Q.
      • Du H.
      • et al.
      Lung ultrasound findings in patients with COVID-19 pneumonia.
      • Poggiali E.
      • Dacrema A.
      • Bastoni D.
      • et al.
      Can lung US help critical care clinicians in the early diagnosis of novel coronavirus (COVID-19) pneumonia?.

       Goals of This Investigation

      In this study, we sought to explore, in a population of adult patients presenting to the ED with symptoms potentially related to SARS-CoV-2 infection, whether the integration of lung ultrasonography with the initial clinical evaluation was associated with increased sensitivity for the diagnosis of COVID-19 pneumonia, therefore facilitating the identification of false-negative RT-PCR results.

      Materials and Methods

       Study Design and Setting

      This was a prospective cohort study held at the ED of the Città della Salute e della Scienza di Torino University Hospital, Turin, Italy, an urban university-affiliated ED with an approximate annual volume of 75,000 visits per year and main site of the University of Turin Emergency Medicine Residency Program. The protocol was approved by the Città della Salute e della Scienza di Torino institutional review board. All patients or their substitute decisionmakers provided informed consent and all data were immediately deidentified. The study was conducted in accordance with the principles of the Declaration of Helsinki for clinical research involving human subjects.

       Selection of Participants

      We considered eligible all adult patients (≥18 years) who, at triage, screened positive for acute symptoms commonly associated with SARS-CoV-2 infection (ie, fever, dyspnea, new or worsening cough, sore throat, diarrhea, ageusia, anosmia, and asthenia) during the study period. Screening-positive patients were assigned to cohorts in a dedicated area of the ED and subsequently evaluated and approached for study enrollment by the treating physician, regardless of their primary language. Patients known to be infected by SARS-CoV-2, requiring an urgent psychiatric assessment, or already intubated at ED arrival were excluded. Conversely, patients who required noninvasive mechanical ventilation were included in the study. Availability of an attending physician with expertise in lung ultrasonography (ie, completion of a lung ultrasonographic training course accredited by the Italian Society of Emergency Medicine
      Italian Society of Emergency Medicine
      Regolamento attività formative - versione del 14/03/2015.
      and performance and interpretation of at least 40 lung ultrasonographic examinations) was also required for enrollment. All physicians received a training module describing study, scanning protocol, and COVID-19/viral pneumonia lung ultrasonographic findings known at study initiation.
      • Peng Q.-Y.
      • Wang X.-T.
      • Zhang L.-N.
      Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.

       Measurements

      Immediately after the initial clinical evaluation (ie, medical history, history of present illness, physical examination, and ECG, but not laboratory analyses and chest radiography), the treating physician (ie, the attending physician responsible for the patient care) postulated and recorded in a brief case report form the likelihood of SARS-CoV-2 pneumonia, expressed as a dichotomous result (SARS-CoV-2 pneumonia yes/no
      • Moors G.
      Exploring the effect of a middle response category on response style in attitude measurement.
      ). As soon as possible after this initial assessment, the same physician performed a lung ultrasonographic examination and recorded a new diagnostic hypothesis on COVID-19 status (SARS-CoV-2 pneumonia yes/no), based on both clinical and lung ultrasonographic findings (“integrated” assessment). At this point, neither RT-PCR nor chest radiography or CT scan results were available yet. On the contrary, for 143 of 228 patients, the results of arterial blood gas tests were available (Table E1, available online at http://www.annemergmed.com).
      For the lung ultrasonographic examination, either a curvilinear transducer (5 to 3 MHz) or a cart-based ultrasonographic system (Esaote Mylab5, Genova, Italy) with abdominal preset or a handheld device (Butterfly IQ; Butterfly Network Inc., Guiford, CT) with lung preset (3 MHz) was used. We implemented a 12-zone (6 areas per hemithorax) scanning protocol, as previously described
      • Bouhemad B.
      • Brisson H.
      • Le-Guen M.
      • et al.
      Bedside ultrasound assessment of positive end-expiratory pressure–induced lung recruitment.
      (Figure 1). Briefly, we divided each hemithorax into anterior, lateral, and posterior zones, and upper and lower zones (divided by the internipple line); anterior aspect of the chest was identified by the anterior axillary line, lateral aspect of the chest by the anterior and posterior axillary lines, and posterior aspect of the chest by the posterior axillary line and the spine, not including the scapular area.
      • Volpicelli G.
      • Elbarbary M.
      • Blaivas M.
      • et al.
      International evidence-based recommendations for point-of-care lung ultrasound.
      ,
      • Volpicelli G.
      • Mussa A.
      • Garofalo G.
      • et al.
      Bedside lung ultrasound in the assessment of alveolar-interstitial syndrome.
      ,
      • Jambrik Z.
      • Monti S.
      • Coppola V.
      • et al.
      Usefulness of ultrasound lung comets as a nonradiologic sign of extravascular lung water.
      Because sonographic findings of SARS-CoV-2 pneumonia had been described in only small cohorts of patients
      • Peng Q.-Y.
      • Wang X.-T.
      • Zhang L.-N.
      Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.
      ,
      • Lomoro P.
      • Verde F.
      • Zerboni F.
      • et al.
      COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: single-center study and comprehensive radiologic literature review.
      and no validated diagnostic criteria have been published to our knowledge, we decided to use a pragmatic approach, without stringent sonographic definitions for SARS-CoV-2–related pneumonia patterns. In agreement with previously published descriptions of viral pneumonia on lung ultrasonography, presence of focal or diffuse interstitial syndrome associated with spared areas, subpleural consolidations, and irregular or thickened pleural line was considered suggestive of SARS-CoV-2–related pneumonia (Figure E1, available online at http://www.annemergmed.com).
      • Peng Q.-Y.
      • Wang X.-T.
      • Zhang L.-N.
      Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.
      ,
      • Lomoro P.
      • Verde F.
      • Zerboni F.
      • et al.
      COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: single-center study and comprehensive radiologic literature review.
      • Zhang Y.
      • Li J.
      • Yang J.
      • et al.
      Lung ultrasonography for the diagnosis of 11 patients with acute respiratory distress syndrome due to bird flu H7N9 infection.
      • Peris A.
      • Zagli G.
      • Barbani F.
      • et al.
      The value of lung ultrasound monitoring in H1N1 acute respiratory distress syndrome.
      After completion of the lung ultrasonographic examination, all patients also underwent chest imaging (chest radiography, CT, or both) according to clinical indication.
      Figure thumbnail gr1
      Figure 1Lung ultrasonographic scanning protocol. Areas 1, 2, 5, and 6 represent right and left anterior superior and inferior zones, respectively; areas 3, 4, 7, and 8, right and left lateral superior and inferior zones, respectively; and areas 9, 10, 11, and 12, right and left posterior superior and inferior zones, respectively.
      As part of study protocol, the treating physician was asked to record in the case report form the time between the end of the clinical evaluation and the beginning of lung ultrasonography, and the time required to complete the 12-zone lung ultrasonographic scanning.

       Outcomes

      For all patients, SARS-CoV-2 detection was performed at first on nasopharyngeal swab samples; briefly, after purification with a QIA symphony DSP virus/pathogen kit (QIAGEN, Venlo, the Netherlands), RT-PCR was performed with the GeneFinder 2019 nCoVRealAmp Kit (Elitech, Puteaux, France) or the Simplexa COVID-19 Direct kit (Diasorin Molecular, Cypress, CA), following manufacturer’s instructions. All RT-PCR–positive cases were considered true positive.
      WHO Global Surveillance for human infection with coronavirus disease (COVID-19).
      In patients with a negative RT-PCR result, a second nasopharyngeal swab was repeated when clinical, sonographic, laboratory, or imaging evaluations were suspicious for SARS-CoV-2 infection. In addition, in some of these inconclusive cases, other diagnostic tests such as RT-PCR for other respiratory viruses, urinary antigens for Streptococcus pneumoniae and Legionella pneumophila, bronchoalveolar lavage microbiological studies (including SARS-CoV-2 RT-PCR), and chest CT (if not yet performed) were also performed according to the clinician’s indication. Patients with an initial SARS-CoV-2 RT-PCR–negative result were classified as COVID-19 positive if a second RT-PCR test (performed within 72 hours from the initial assessment), either on a nasopharyngeal swab or bronchoalveolar lavage, yielded a positive result. Conversely, patients with results for initial SARS-CoV-2 RT-PCR and all other tests (ie, clinical evaluation, lung ultrasonography, laboratory, and imaging) concordantly negative were classified as having a true COVID-19–negative case and had no further microbiological testing. For all patients, follow-up was performed at 30 days after the initial assessment either by telephone (if patients had been discharged) or in person (for patients still hospitalized).

       Analysis

      Descriptive results are presented as median and interquartile range (IQR) for continuous data, or as number and percentage for ordinal data.
      The diagnostic accuracies of clinical evaluation alone, clinical–lung ultrasonographic integrated assessment, and first RT-PCR test were calculated through sensitivity, specificity, positive predictive values and negative predictive values, positive and negative likelihood ratios, and area under the receiver operating characteristic curve.
      • DeLong E.R.
      • DeLong D.M.
      • Clarke-Pearson D.L.
      Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
      We compared the difference in accuracies by using McNemar’s test for paired data.
      • McNemar Q.
      Note on the sampling error of the difference between correlated proportions or percentages.
      Net reclassification index
      • Pencina M.J.
      • D’Agostino R.B.
      • D’Agostino R.B.
      • et al.
      Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.
      was used to evaluate the clinical usefulness of the different assessments and diagnostic tests.
      We estimated that a sample size of 228 patients was necessary to achieve a power of 90% to observe an absolute difference of 8% in sensitivity between clinical–lung ultrasonographic integrated evaluation and RT-PCR, with 1% α error (one-sided test).
      Data were collected with an Excel (version 16.43; Microsoft, Redmond, WA) spreadsheet, and analyses were conducted with Stata (version 13.0/SE; StataCorp, College Station, TX).

      Results

      Between April 1 and April 20, 1,376 consecutive patients presented to the ED of the Città della Salute e della Scienza di Torino University Hospital; 451 patients were considered eligible, and 223 fulfilled exclusion criteria; 228 provided informed consent and were enrolled in the study (Figure 2). Tables 1 and E1 (available online at http://www.annemergmed.com) summarize baseline characteristics of enrolled patients by infection status. Twenty physicians participated in the study, enrolling a median number of 12 patients each (IQR 5 to 13 patients) (Table E2, available online at http://www.annemergmed.com).
      Figure thumbnail gr2
      Figure 2Standards for Reporting Diagnostic Accuracy diagram of enrolled patients, grouped by COVID-19 infection status according to clinical and integrated assessments and RT-PCR results. C-LUS, Clinical–lung ultrasonographic integrated assessment.
      Table 1Baseline characteristics and ED outcome of enrolled patients according to COVID-19 infection status.
      No COVID-19 (N=121)COVID-19 (N=107)Total (N=228)
      Age, median (IQR), y50.3 (38.9–63.3)62.8 (50.5–77.6)57.7 (43.3–71.2)
       Men, No. (%)52 (43.7)59 (54.1)111 (48.7)
       Women, No. (%)67 (56.3)50 (45.9)117 (51.3)
       Days from onset of symptoms, median (IQR)4 (2–10)6 (3–7)6 (2–8)
      ED outcome, No. (%)
       Home discharge95 (78.5)27 (25.2)122 (53.5)
       Ward admission25 (20.7)65 (60.8)90 (39.5)
       ICU/HDU admission1 (0.8)8 (7.5)9 (3.9)
       ED death07 (6.5)7 (3.1)
      Symptoms, No. (%)
       Ageusia8 (6.6)15 (14.0)23 (10.1)
       Anosmia6 (5.0)12 (11.2)18 (7.9)
       Cough53 (43.8)53 (49.5)106 (46.5)
       Diarrhea26 (21.5)15 (14.0)41 (18.0)
       Fatigue22 (18.3)27 (25.2)49 (21.6)
       Fever68 (56.2)79 (73.8)147 (64.5)
       Headache9 (7.4)2 (1.9)11 (4.8)
       Shortness of breath40 (33.6)40 (37.4)80 (35.1)
       Sore throat21 (17.4)5 (4.7)26 (11.4)
      Comorbidities, No. (%)
       Atrial fibrillation/flutter9 (7.4)15 (14)24 (10.5)
       Asthma7 (5.8)2 (1.9)9 (4)
       CAD8 (6.6)18 (16.8)26 (11.4)
       Cancer
      History of active or inactive solid or hematologic neoplasm.
      13 (10.7)18 (16.8)31 (13.8)
       Cerebrovascular disease10 (8.3)8 (7.5)18 (7.9)
       Chronic kidney disease2 (1.7)5 (4.7)7 (3.1)
       COPD8 (6.6)12 (11.2)20 (8.8)
       Diabetes mellitus5 (4.1)19 (17.8)24 (10.5)
       DVT/PE2 (1.7)4 (3.8)6 (2.6)
       Hypertension31 (25.6)49 (45.8)80 (35.1)
      Smoking history
       Active16 (13.2)8 (7.5)24 (10.5)
       Inactive14 (11.6)17 (15.9)31 (13.6)
      HDU, High dependency unit; ICU, Intensive Care Unit; CAD, coronary artery disease; COPD, chronic obstructive pulmonary disease; DVT, deep venous thrombosis; PE, pulmonary embolism.
      Patients were considered as having suspected COVID-19 infection when reporting at least one of the following symptoms: fever, dyspnea, cough, sore throat, diarrhea, ageusia, anosmia, or asthenia.
      History of active or inactive solid or hematologic neoplasm.
      The median time between the end of the clinical evaluation and start of lung ultrasonography was 10 minutes for all cases (IQR 5 to 10 minutes and 5 to 15 minutes for SARS-CoV-2–negative and –positive patients, respectively).
      The diagnostic accuracies of the clinical evaluation alone, clinical–lung ultrasonographic integrated assessment, and RT-PCR are shown in Tables 2 and 3. As described in the “Materials and Methods” section, RT-PCR was considered the reference test in the case of a positive result. Therefore, by protocol, this test did not provide false-positive results (ie, RT-PCR–positive patients were considered to have true-positive cases). The clinical–lung ultrasonographic integrated assessment had higher sensitivity than the first nasopharyngeal RT-PCR (94.4% [95% confidence interval {CI} 88.2% to 97.9%] versus 80.4% [95% CI 71.6% to 87.4%]); the clinical–lung ultrasonographic integrated assessment also demonstrated excellent specificity (95% [95% CI 89.5% to 98.2%]).
      Table 2Two-by-two table of clinical–lung ultrasonographic integrated evaluation and RT-PCR.
      C-LUS IntegratedPositiveNegativeTotalRT-PCRPositiveNegativeTotal
      COVID-191016107COVID-198621107
      Non–COVID-196115121Non–COVID-190121121
      Total107121228Total86142228
      Table 3Diagnostic accuracy of clinical evaluation only, clinical evaluation integrated with lung ultrasonographic findings, and RT-PCR.
      Sensitivity (95% CI)Specificity (95% CI)Positive Predictive Value (95% CI)Negative Predictive Value (95% CI)Positive Likelihood Ratio (95% CI)Negative Likelihood Ratio (95% CI)
      Clinical evaluation only81.3 (72.6–88.2)63.6 (54.4–72.2)66.4 (57.6–74.4)79.4 (70–86.9)2.24 (1.7–2.9)0.29 (0.19–0.45)
      C-LUS integrated assessment94.4 (88.2–97.9)95 (89.5–98.2)94.4 (88.2–97.9)95 (89.5–98.2)19 (8.7–41.6)0.06 (0.03–0.13)
      RT-PCR80.4 (71.6–87.4)100
      RT-PCR specificity was 100% by study design (no CIs estimated).
      100
      RT-PCR specificity was 100% by study design (no CIs estimated).
      85.2 (78.3–90.6)NA0.20 (0.13–0.29)
      NA, Not applicable.
      RT-PCR specificity was 100% by study design (no CIs estimated).
      Figure 2 shows the distribution of patients according to the clinical evaluation alone, the clinical evaluation integrated with lung ultrasonography, the results of the first RT-PCR test, and the final COVID-19 status.
      Of the 142 patients who tested negative at the first nasopharyngeal RT-PCR, 51 had a second RT-PCR test, either on nasopharyngeal swab or bronchoalveolar lavage, when clinical, sonographic, laboratory, or imaging evaluation results were suspicious for SARS-CoV-2 infection. Twenty-one patients (14.8%) of those who tested negative at the first nasopharyngeal RT-PCR test had a positive result at a subsequent molecular test performed within 72 hours, and were therefore classified as having false-negative cases. All of them were correctly identified by the integrated clinical–lung ultrasonographic assessment. In particular, 17 patients were identified as highly likely to be SARS-CoV-2 positive by both clinical alone and clinical–lung ultrasonographic integrated assessments; for another 4 patients, the clinical–lung ultrasonographic integrated assessment but not the clinical assessment alone correctly suggested COVID-19 infection.
      On the other hand, the integrated evaluation falsely identified 6 patients as possibly infected by SARS-CoV-2 (false-positive result) and 6 other infected patients as unlikely to have COVID-19 pneumonia (false-negative result). The clinical characteristics, final diagnoses, and lung ultrasonographic findings of the 6 patients identified as false positives at the integrated clinical-lung ultrasonographic evaluation are reported in the Table E3, available online at http://www.annemergmed.com.
      In addition, for some of the patients who tested negative at the first nasopharyngeal RT-PCR test and for whom the clinical suspicion of SARS-CoV-2 infection remained high, other diagnostic tests such as RT-PCR for other respiratory viruses (n=13), urinary antigens for S pneumoniae and L pneumophila (n=32), and chest CT (if not yet performed; n=39) were also performed according to the clinician’s indication.
      The patients with initial RT-PCR and all other tests (ie, clinical evaluation, lung ultrasonography, laboratory, and imaging) not suggestive of SARS-CoV-2 infection were classified as having true COVID-19–negative cases. We were able to follow up all enrolled patients at 30 days. Among those patients classified in the ED as being SARS-CoV-2 negative, no one received a diagnosis of infection or reported symptoms or events consistent with it.
      Net reclassification index for positive COVID-19 infections was 14.0% (95% CI 8.1% to 22.1%), whereas that for negative cases was –4.9% (95% CI –1.9% to 10.7%); this estimate is an upper bound of the magnitude of the net reclassification index because, by design, there were no false-positive results for RT-PCR.
      The median duration of the lung ultrasonographic evaluation was 5 minutes (IQR 5 to 8 minutes and 5 to 10 minutes for patients with SARS-CoV-2–negative and –positive results, respectively).

      Limitations

      The absence of a true criterion standard for the diagnosis of SARS-CoV-2 infection is a limitation of our study. We assumed that patients who tested positive at RT-PCR had true-positive cases because only occasional false-positive results have been described.
      • Sethuraman N.
      • Jeremiah S.S.
      • Ryo A.
      Interpreting diagnostic tests for SARS-CoV-2.
      ,
      SARS-CoV-2 molecular assay evaluation: results.
      However, as previously discussed, false-negative results with RT-PCR are not uncommon. In our study, an incorporation bias (and possibly a verification bias) may have occurred for patients with negative results for RT-PCR and clinical–lung ultrasonographic integrated assessment and for whom further investigations were not pursued. We could not perform multiple RT-PCR analyses or lung CT scans in patients with lower clinical suspicion, laboratory results not suggestive of SARS-CoV-2 pneumonia, and normal chest radiography results. However, because the bias occurred for both RT-PCR and clinical–lung ultrasonographic integrated assessment, it might have overestimated both sensitivities but was unlikely to affect one more than the other. Moreover, at the 30-day follow-up, none of the patients with negative cases reported symptoms or events consistent with SARS-CoV-2 infection.
      Several factors may have led to an overestimation of the diagnostic accuracy of the clinical–lung ultrasonographic integrated assessment in our study or to limitations in its generalizability. First was institutional and provider expertise. The study was conducted in a tertiary academic hospital with extensive experience in the use of point-of-care ultrasonography in the ED; also, the presence of a physician expert in lung ultrasonography (>40 studies) was essential for enrollment. Although lung ultrasonographic evaluation is rapidly increasing in the ED, many centers have not systematically implemented its use in their diagnostic algorithms. Moreover, 12-zone lung ultrasonography is a relatively advanced and time-consuming scanning protocol. The diagnostic accuracy of our clinical–lung ultrasonographic integrated assessment may therefore not be as high in centers less experienced and comfortable with this technique. However, lung ultrasonography has already been shown to be easy to learn and to require a relatively small number of studies to achieve competency.
      • See K.C.
      • Ong V.
      • Wong S.H.
      • et al.
      Lung ultrasound training: curriculum implementation and learning trajectory among respiratory therapists.
      ,
      • Rouby J.-J.
      • Arbelot C.
      • Gao Y.
      • et al.
      Training for lung ultrasound score measurement in critically ill patients.
      Second was the “unique” recruitment period. Our study was conducted during a short (20-day), single-season, and epidemiologically unparalleled period. Therefore, the utility of lung ultrasonography for the diagnosis of SARS-CoV-2 infection and other viral pneumonias27 outside an outbreak scenario (characterized by high prevalence of a single type of infection) needs to be evaluated in further studies encompassing longer recruitment periods. Third was the cohort strategy. All enrolled patients were evaluated in a dedicated area of the ED to minimize the risk of cross-contamination. This cohort strategy may have led to an overestimation of physicians’ clinical suspicion for SARS-CoV-2 infection. In addition, performance of the initial clinical assessment and lung ultrasonography by the same physician could have introduced a behavioral confirmation bias (ie, tendency to look for and interpret one’s prior belief). Fourth was the lack of prespecified lung ultrasonographic diagnostic criteria. At the performance of the study, limited description of sonographic findings of SARS-CoV-2 pneumonia was available
      • Peng Q.-Y.
      • Wang X.-T.
      • Zhang L.-N.
      Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.
      ,
      • Lomoro P.
      • Verde F.
      • Zerboni F.
      • et al.
      COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: single-center study and comprehensive radiologic literature review.
      and therefore we decided not to use stringent sonographic definitions. Paradoxically, this pragmatic approach may become a limitation to generalizability when one considers that many ED providers do not have the same experience and expertise in lung ultrasonography as our group has. Fifth was the risk of verification bias. The present reference test for evaluating accuracy of the clinical assessment alone and integrated with lung ultrasonographic evaluation is a test known to be imperfect; specifically, for ruling out COVID-19 infection. In the subgroup of patients with an initially negative result with RT-PCR, we added additional tests in case of a high suspicion of disease to reduce the possible number of true-positive cases classified as false-positive ones. Moreover, we also followed up enrolled patients to confirm disease status. Sixth, our population included only adult patients. Although our results cannot be directly translated to the pediatric population in the absence of further studies, feasibility of lung ultrasonography in children with COVID-19 infection has been reported.
      • Musolino A.M.
      • Supino M.C.
      • Buonsenso D.
      • et al.
      Lung ultrasound in children with COVID-19: preliminary findings.

      Discussion

      Our study demonstrates that, in adult patients presenting to the ED with symptoms potentially related to SARS-CoV-2 infection, an integrated clinical and pulmonary sonographic approach has high sensitivity and specificity for COVID-19 pneumonia and may therefore help to identify false-negative results occurring with nasopharyngeal specimens analyzed with RT-PCR. To our knowledge, this is the first study to demonstrate, in a cohort of patients with suspected SARS-CoV-2 infection, an improvement of sensitivity with the implementation of lung ultrasonography with clinical evaluation.
      Although in laboratory settings SARS-CoV-2 RT-PCR has been shown to be extremely sensitive and specific,
      SARS-CoV-2 molecular assay evaluation: results.
      actual clinical sensitivity of RT-PCR has been reported to be significantly lower,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      likely related to multiple factors, including sampling errors or inappropriate timing of sample collection (either too early or too late in the clinical course of the disease
      SARS-CoV-2 molecular assay evaluation: results.
      ,
      • Wang W.
      • Xu Y.
      • Gao R.
      • et al.
      Detection of SARS-CoV-2 in different types of clinical specimens.
      ). In our study, clinical sensitivity was similar to or even higher than that previously reported,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      ,
      • Wang W.
      • Xu Y.
      • Gao R.
      • et al.
      Detection of SARS-CoV-2 in different types of clinical specimens.
      ,
      • Long C.
      • Xu H.
      • Shen Q.
      • et al.
      Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT?.
      although a recently published meta-analysis comparing chest CT and initial RT-PCR sensitivity suggested higher sensitivity (89% [95% CI 81% to 94%]).
      • Kim H.
      • Hong H.
      • Yoon S.H.
      Diagnostic performance of CT and reverse transcriptase–polymerase chain reaction for coronavirus disease 2019: a meta-analysis.
      However, in this meta-analysis, substantial heterogeneity was identified (I2=90%), most of the studies had retrospective design, details on RT-PCR procedures were not provided, and often the reference standard was considered the RT-PCR itself.
      • Long C.
      • Xu H.
      • Shen Q.
      • et al.
      Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT?.
      Nevertheless, even with RT-PCR sensitivity as high as 89%,
      • Kim H.
      • Hong H.
      • Yoon S.H.
      Diagnostic performance of CT and reverse transcriptase–polymerase chain reaction for coronavirus disease 2019: a meta-analysis.
      the number of false-negative results could be unacceptable,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      with substantial risk of disease transmission and further propagation of the infection in the community and health care settings.
      • West C.P.
      • Montori V.M.
      • Sampathkumar P.
      COVID-19 testing: the threat of false-negative results.
      ,
      • Bai Y.
      • Yao L.
      • Wei T.
      • et al.
      Presumed asymptomatic carrier transmission of COVID-19.
      For example, in our cohort of patients, with prevalence of COVID-19 infection of 46.9% and RT-PCR sensitivity of 80.4%, a negative RT-PCR result would still carry a posttest probability of having the infection of 15%,
      • Woloshin S.
      • Patel N.
      • Kesselheim A.S.
      False negative tests for SARS-CoV-2 infection—challenges and implications.
      whereas the clinical–lung ultrasonographic integrated assessment, with a sensitivity of 94.4%, would reduce it to 4.9%.
      The use of a clinical–lung ultrasonographic integrated approach may be able to improve current limitations of molecular diagnostic testing in patients presenting to the ED with COVID-19 symptoms. Our data demonstrate that our integrated approach improves the sensitivity of the first RT-PCR test performed on the nasopharyngeal sample collected at assessment in the ED, and allows the correct reclassification of nearly 15% of patients who otherwise would have been wrongly identified as having a negative result by RT-PCR. Still, the diagnostic accuracy of the clinical–lung ultrasonographic integrated assessment, although high, cannot justify its use as the sole diagnostic tool to rule out SARS-CoV-2 infection in patients presenting to the ED with symptoms compatible with COVID-19 pneumonia. The smaller, but not negligible, number of false-negative results (n=6) observed with this approach would have exposed other patients and health care workers or people in the community to a considerable risk of disease transmission. Therefore, we suggest the use of the proposed clinical and sonographic integrated approach as a method to better identify potential false-negative nasopharyngeal swab RT-PCR results. Patients with symptoms associated with SARS-CoV-2 infection and lung ultrasonographic findings of viral pneumonia should have further investigations in the presence of negative RT-PCR result at the nasopharyngeal swab, and isolation precautions should be maintained.
      In our study, the integrated clinical–lung ultrasonographic evaluation falsely identified 6 patients as possibly infected by SARS-CoV-2 (false positive) and 6 other infected patients as unlikely to have COVID-19 pneumonia (false negative). False-negative cases could have been missed because it was too early in the course of the disease or because the patients did not develop pneumonia. On the contrary, false-positive cases, whose clinical characteristics, final diagnoses, and lung ultrasonographic findings are detailed in Table E3 (available online at http://www.annemergmed.com), were all affected by diseases that could mimic the pathologic findings detected by lung ultrasonography in COVID-19 patients, leading the treating physician, during the pandemic outbreak, to choose COVID-19 as the most likely final diagnosis. It is well known that lung ultrasonography alone is able to discriminate neither different forms of diffuse interstitial syndrome (eg, interstitial lung disease, acute decompensate heart failure, acute respiratory distress syndrome, interstitial pneumonia) nor of lung parenchymal consolidations. On the contrary, the integration of lung ultrasonography with clinical data would increase its diagnostic accuracy in all these conditions.
      • Pivetta E.
      • Goffi A.
      • Nazerian P.
      • et al.
      Lung ultrasound integrated with clinical assessment for the diagnosis of acute decompensated heart failure in the emergency department: a randomized controlled trial.
      ,
      • Volpicelli G.
      • Mussa A.
      • Garofalo G.
      • et al.
      Bedside lung ultrasound in the assessment of alveolar-interstitial syndrome.
      The ability to perform lung ultrasonography at the bedside, the short time required to complete it, the lower costs, and the absence of radiation exposure represent other significant strengths of our integrated approach when compared with other proposed diagnostic strategies, such as integration with chest CT scan.
      • Xie X.
      • Zhong Z.
      • Zhao W.
      • et al.
      Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing.
      ,
      • Tavare A.N.
      • Braddy A.
      • Brill S.
      • et al.
      Managing high clinical suspicion COVID-19 inpatients with negative RT-PCR: a pragmatic and limited role for thoracic CT.
      In our study, the median duration of lung ultrasonographic evaluation was 5 minutes (IQR 5 to 8 minutes and 5 to 10 minutes for SARS-CoV-2–negative and –positive patients, respectively). This duration is consistent with what has been reported in other studies.
      • See K.C.
      • Ong V.
      • Wong S.H.
      • et al.
      Lung ultrasound training: curriculum implementation and learning trajectory among respiratory therapists.
      ,
      • Rouby J.-J.
      • Arbelot C.
      • Gao Y.
      • et al.
      Training for lung ultrasound score measurement in critically ill patients.
      However, providers with limited expertise may take longer to perform the 12-zone lung ultrasonographic examination.
      In conclusion, the results of our study suggest that, in patients presenting to the ED with symptoms potentially related to SARS-CoV-2 infection, the integration of lung ultrasonography with clinical evaluation has high sensitivity and specificity for the diagnosis of COVID-19 pneumonia, and it may help to identify false-negative RT-PCR results.
      The authors acknowledge Butterfly Network Inc (New York, NY) for supporting their ED during the SARS-CoV-2 pandemic with 4 Butterfly iQ devices.

      Appendix

       Molinette MedUrg Group on Lung Ultrasound

      Federico Baldassa, MD, Paolo Baron, MD, Giordano Bianchi, MD, Busso V, Andrea Conterno, MD, Paola Del Rizzo, MD, Paolo Fascio Pecetto, MD, Francesca Giachino, MD, Andrea Iannaccone, MD, Patrizia Ferrera, MD, Franco Riccardini, MD, Claudia Sacchi, MD, Michela Sozzi, MD, Silvia Totaro, MD, Pasqualina Visconti, MD, Francesca Risi, MD, Francesca Basile, MD, Denise Baricocchi, MD, Alessia Beux, MD, Valentina Beux, MD, Paolo Bima, MD, Irene Cara, MD, Liliana Chichizola, MD, Francesca Dellavalle, MD, Federico Grosso, MD, Giulia Labarile, MD, Matteo Oddi, MD, Marco Ottino, MD, Ilaria Pia, MD, Virginia Scategni, MD, Astrid Surra, MD.

      Supplementary Data

      • Tables E1 - E3

        Vital signs and laboratory results of enrolled patients according to COVID-19 infection status. Abbreviations: IQR, interquartile range; LDH, lactate dehydrogenase; SpO2/FiO2, peripheral blood oxygen saturation/fraction of inspired oxygen.

        Table E2. Physicians participating in the study and number of cases they evaluated. Abbreviations: TP, true positives; TN, true negatives; FP, false positives; FN, false negatives

        Table E3. Features of patients identified as false positive by the clinical evaluation integrated with lung ultrasound (LUS). Abbreviations: HFpEF, heart failure with preserved ejection fraction; COPD, Chronic obstructive pulmonary disease; COVID-19: coronavirus disease of 2019.

      Figure thumbnail figs1
      Figure E1LUS findings in COVID-19 positive patients. Panel A: irregular and thickened pleural line (arrow); Panel B: multiple small, subpleural consolidations (arrow); Panel C: B-lines and vertical artifacts (asterisks), they might be present in separate and coalescent forms and arising from a peripheral consolidations and/or from pleural points (Peng QY et al, Intensive Care Med, 2020; Volpicelli G et al, Ultrasound J, 2020).

      References

        • West C.P.
        • Montori V.M.
        • Sampathkumar P.
        COVID-19 testing: the threat of false-negative results.
        Mayo Clin Proc. 2020; 95: 1127-1129
        • Sharfstein J.M.
        • Becker S.J.
        • Mello M.M.
        Diagnostic testing for the novel coronavirus.
        JAMA. 2020; 323: 1437-1438
        • Cheng M.P.
        • Papenburg J.
        • Desjardins M.
        • et al.
        Diagnostic testing for severe acute respiratory syndrome–related coronavirus 2: a narrative review.
        Ann Intern Med. 2020; 172: 726-734
        • Marcotte L.M.
        • Liao J.M.
        Incorporating test characteristics into SARS-CoV-2 testing policy—sense and sensitivity.
        (Available at:)
        • Woloshin S.
        • Patel N.
        • Kesselheim A.S.
        False negative tests for SARS-CoV-2 infection—challenges and implications.
        N Engl J Med. 2020; 383: e38
      1. COVID-19: EU recommendations for testing strategies.
        (Available at:)
        • European Centre for Disease Prevention and Control
        An overview of the rapid test situation for COVID-19 diagnosis in the EU/EEA: technical report, April 1st, 2020.
        (Available at:)
        • Sethuraman N.
        • Jeremiah S.S.
        • Ryo A.
        Interpreting diagnostic tests for SARS-CoV-2.
        JAMA. 2020; 323: 2249-2251
        • Winichakoon P.
        • Chaiwarith R.
        • Liwsrisakun C.
        • et al.
        Negative nasopharyngeal and oropharyngeal swabs do not rule out COVID-19.
        J Clin Microbiol. 2020; 58 (e00297-20)
        • Xie X.
        • Zhong Z.
        • Zhao W.
        • et al.
        Chest CT for typical 2019-nCoV pneumonia: relationship to negative RT-PCR testing.
        Radiology. 2020; 296: E41-E45
        • Centers for Disease Control and Prevention
        Algorithm to assist in the interpretation of influenza testing results and clinical decision-making during periods when influenza viruses are circulating in the community.
        (Available at:)
        • Centers for Disease Control and Prevention
        Interim infection prevention and control recommendations for patients with suspected or confirmed coronavirus disease 2019 (COVID-19) in healthcare settings.
        (Available at:) (Updated April 13, 2020. Accessed May 2, 2020)
        • Centers for Disease Control and Prevention
        Discontinuation of transmission-based precautions and disposition of patients with COVID-19 in healthcare settings (interim guidance).
        (Available at:)
        • Lichtenstein D.
        • MéZièRe G.
        • Biderman P.
        • et al.
        The comet-tail artifact: an ultrasound sign of alveolar-interstitial syndrome.
        Am J Respir Crit Care Med. 1997; 156: 1640-1646
        • Pivetta E.
        • Goffi A.
        • Nazerian P.
        • et al.
        Lung ultrasound integrated with clinical assessment for the diagnosis of acute decompensated heart failure in the emergency department: a randomized controlled trial.
        Eur J Heart Fail. 2019; 21: 754-766
        • Volpicelli G.
        • Elbarbary M.
        • Blaivas M.
        • et al.
        International evidence-based recommendations for point-of-care lung ultrasound.
        Intensive Care Med. 2012; 38: 577-591
        • Maw A.M.
        • Hassanin A.
        • Ho P.M.
        • et al.
        Diagnostic accuracy of point-of-care lung ultrasonography and chest radiography in adults with symptoms suggestive of acute decompensated heart failure: a systematic review and meta-analysis.
        JAMA Netw Open. 2019; 2e190703
        • Peng Q.-Y.
        • Wang X.-T.
        • Zhang L.-N.
        Chinese Critical Care Ultrasound Study Group (CCUSG). Findings of lung ultrasonography of novel corona virus pneumonia during the 2019-2020 epidemic.
        Intensive Care Med. 2020; 46: 849-850
        • Xing C.
        • Li Q.
        • Du H.
        • et al.
        Lung ultrasound findings in patients with COVID-19 pneumonia.
        Crit Care. 2020; 24: 174
        • Poggiali E.
        • Dacrema A.
        • Bastoni D.
        • et al.
        Can lung US help critical care clinicians in the early diagnosis of novel coronavirus (COVID-19) pneumonia?.
        Radiology. 2020; 295: E6
        • Italian Society of Emergency Medicine
        Regolamento attività formative - versione del 14/03/2015.
        (Available at:)
        • Moors G.
        Exploring the effect of a middle response category on response style in attitude measurement.
        Qual Quant. 2008; 42: 779-794
        • Bouhemad B.
        • Brisson H.
        • Le-Guen M.
        • et al.
        Bedside ultrasound assessment of positive end-expiratory pressure–induced lung recruitment.
        Am J Respir Crit Care Med. 2011; 183: 341-347
        • Volpicelli G.
        • Mussa A.
        • Garofalo G.
        • et al.
        Bedside lung ultrasound in the assessment of alveolar-interstitial syndrome.
        Am J Emerg Med. 2006; 24: 689-696
        • Jambrik Z.
        • Monti S.
        • Coppola V.
        • et al.
        Usefulness of ultrasound lung comets as a nonradiologic sign of extravascular lung water.
        Am J Cardiol. 2004; 93: 1265-1270
        • Lomoro P.
        • Verde F.
        • Zerboni F.
        • et al.
        COVID-19 pneumonia manifestations at the admission on chest ultrasound, radiographs, and CT: single-center study and comprehensive radiologic literature review.
        Eur J Radiol Open. 2020; 7: 100231
        • Zhang Y.
        • Li J.
        • Yang J.
        • et al.
        Lung ultrasonography for the diagnosis of 11 patients with acute respiratory distress syndrome due to bird flu H7N9 infection.
        Virol J. 2015; 12: 176
        • Peris A.
        • Zagli G.
        • Barbani F.
        • et al.
        The value of lung ultrasound monitoring in H1N1 acute respiratory distress syndrome.
        Anaesthesia. 2010; 65: 294-297
      2. WHO Global Surveillance for human infection with coronavirus disease (COVID-19).
        (Available at:)
        • DeLong E.R.
        • DeLong D.M.
        • Clarke-Pearson D.L.
        Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
        Biometrics. 1988; 44: 837-845
        • McNemar Q.
        Note on the sampling error of the difference between correlated proportions or percentages.
        Psychometrika. 1947; 12: 153-157
        • Pencina M.J.
        • D’Agostino R.B.
        • D’Agostino R.B.
        • et al.
        Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond.
        Stat Med. 2008; 27 (discussion 207-212): 157-172
      3. SARS-CoV-2 molecular assay evaluation: results.
        (Available at:)
        • See K.C.
        • Ong V.
        • Wong S.H.
        • et al.
        Lung ultrasound training: curriculum implementation and learning trajectory among respiratory therapists.
        Intensive Care Med. 2016; 42: 63-71
        • Rouby J.-J.
        • Arbelot C.
        • Gao Y.
        • et al.
        Training for lung ultrasound score measurement in critically ill patients.
        Am J Respir Crit Care Med. 2018; 198: 398-401
        • Musolino A.M.
        • Supino M.C.
        • Buonsenso D.
        • et al.
        Lung ultrasound in children with COVID-19: preliminary findings.
        Ultrasound Med Biol. 2020; 46: 2094-2098
        • Wang W.
        • Xu Y.
        • Gao R.
        • et al.
        Detection of SARS-CoV-2 in different types of clinical specimens.
        JAMA. 2020; 323: 1843-1844
        • Long C.
        • Xu H.
        • Shen Q.
        • et al.
        Diagnosis of the coronavirus disease (COVID-19): rRT-PCR or CT?.
        Eur J Radiol. 2020; 126: 108961
        • Kim H.
        • Hong H.
        • Yoon S.H.
        Diagnostic performance of CT and reverse transcriptase–polymerase chain reaction for coronavirus disease 2019: a meta-analysis.
        Radiology. 2020; 296: E145-E155
        • Bai Y.
        • Yao L.
        • Wei T.
        • et al.
        Presumed asymptomatic carrier transmission of COVID-19.
        JAMA. 2020; 323: 1406-1407
        • Tavare A.N.
        • Braddy A.
        • Brill S.
        • et al.
        Managing high clinical suspicion COVID-19 inpatients with negative RT-PCR: a pragmatic and limited role for thoracic CT.
        Thorax. 2020; 75: 537-538