Advertisement

In Reply:

      We appreciate Dr. Jiang’s interest in our research and the thoughtful comments related to patient selection, model calibration, and performance evaluation metrics. We described the development and evaluation of machine-learning models to predict short-term risk for acute kidney injury in the emergency department (ED) setting.
      • Martinez D.A.
      • Levin S.R.
      • Klein E.Y.
      • et al.
      Early prediction of acute kidney injury in the emergency department with machine-learning methods applied to electronic health record data. Ann Emerg Med.
      This study serves as a proof of concept that electronic health record data, collected very early in the ED encounter, can be used to generate reliable estimations of short-term risk for kidney injury. Our ultimate objective is to incorporate model-based predictions into a clinical decision support (CDS) system that enables better management and outcomes for patients with or at risk for acute kidney injury. Within the context of a complete CDS system, predictive data will be used in combination with surveillance and detection algorithms to target patients with or at risk for acute kidney injury with renally protective CDS.
      In regard to patient selection, we concede that all models derived from electronic health record data are subject to some element of selection bias. However, we believe the major limitation of our model is selection based on presence of outcome data (eg, repeated serum creatinine [sCr] measurement), as discussed in our original “Limitations” section.
      • Martinez D.A.
      • Levin S.R.
      • Klein E.Y.
      • et al.
      Early prediction of acute kidney injury in the emergency department with machine-learning methods applied to electronic health record data. Ann Emerg Med.
      Selection based on sCr measurement during the ED encounter is less problematic because this occurred for greater than 98% of patients hospitalized from our study sites. As pointed out by Dr. Jiang, our use of sCr measured on ED arrival to establish baseline introduces potential for inclusion of patients with community-acquired acute kidney injury and could lead to underestimation of overall acute kidney injury incidence. Although imperfect, this approach is commonly used for estimation of hospital-acquired acute kidney injury incidence and development of hospital-based acute kidney injury prediction models.
      • Hodgson L.E.
      • Sarnowski A.
      • Roderick P.J.
      • et al.
      Systematic review of prognostic prediction models for acute kidney injury (AKI) in general hospital populations.
      ,
      • Wang H.E.
      • Muntner P.
      • Chertow G.M.
      • et al.
      Acute kidney injury and mortality in hospitalized patients.
      Alternative approaches exist, including imputation of baseline using the Modification of Diet in Renal Disease equation as suggested. This is also problematic because it may overestimate acute kidney injury in patients with chronic kidney disease. Exclusion of all patients with elevated sCr levels, also suggested, would preclude identification of acute kidney injury in those with chronic disease, a particularly at-risk population. We posit that these challenges are best addressed through the development of CDS platforms that leverage prediction and detection algorithms simultaneously, as we intend to do.
      We also acknowledge the concern raised about the study’s emphasizing discriminative performance over calibration. Although maximization of calibration can lead to degradation of discrimination,
      • Diamond G.A.
      What price perfection? calibration and discrimination of clinical prediction models.
      we agree these must be balanced and do always consider calibration before model deployment. Our focus on discriminative performance is driven by our intention to generate clear treatment recommendations when risk of acute kidney injury exceeds a threshold, rather than display probabilistic estimates directly to clinician-users. The latter are more affected by calibration error. Precision-recall curves, as suggested by Dr. Jiang, may be the optimal approach for relating the model to clinical utility, especially given the class imbalance (ie, low acute kidney injury incidence) in our data set. This is exactly what was done. Multiple operating points and associated precision-recall curves for predicting stage 1 acute kidney injury were reported in the original submission’s supplementary data.
      • Martinez D.A.
      • Levin S.R.
      • Klein E.Y.
      • et al.
      Early prediction of acute kidney injury in the emergency department with machine-learning methods applied to electronic health record data. Ann Emerg Med.
      Although sensitivity and specificity declined as the prediction horizon widened, classifiers’ precision increased in parallel with acute kidney injury’s cumulative incidence.

      References

        • Martinez D.A.
        • Levin S.R.
        • Klein E.Y.
        • et al.
        Early prediction of acute kidney injury in the emergency department with machine-learning methods applied to electronic health record data. Ann Emerg Med.
        • Hodgson L.E.
        • Sarnowski A.
        • Roderick P.J.
        • et al.
        Systematic review of prognostic prediction models for acute kidney injury (AKI) in general hospital populations.
        BMJ Open. 2017; 7e016591
        • Wang H.E.
        • Muntner P.
        • Chertow G.M.
        • et al.
        Acute kidney injury and mortality in hospitalized patients.
        Am J Nephrol. 2012; 35: 349-355
        • Diamond G.A.
        What price perfection? calibration and discrimination of clinical prediction models.
        J Clin Epidemiol. 1992; 45: 85-89

      Linked Article

      • Selection Bias in the Predictive Analytics With Machine-Learning Algorithm
        Annals of Emergency MedicineVol. 77Issue 2
        • Preview
          I read with great interest the study by Martinez et al,1 who developed a machine-learning model for the early prediction of acute kidney injury. The model was created in the emergency department (ED) and can predict acute kidney injury development at 24, 48, and 72 hours after the index ED visit. The model showed good performance in terms of discrimination. However, model calibration was not performed, leaving unknown whether the model predicts equally well at various risk strata.
        • Full-Text
        • PDF
      • Early Prediction of Acute Kidney Injury in the Emergency Department With Machine-Learning Methods Applied to Electronic Health Record Data
        Annals of Emergency MedicineVol. 76Issue 4
        • Preview
          Acute kidney injury occurs commonly and is a leading cause of prolonged hospitalization, development and progression of chronic kidney disease, and death. Early acute kidney injury treatment can improve outcomes. However, current decision support is not able to detect patients at the highest risk of developing acute kidney injury. We analyzed routinely collected emergency department (ED) data and developed prediction models with capacity for early identification of ED patients at high risk for acute kidney injury.
        • Full-Text
        • PDF