Machine Learning Model for Early Sepsis Risk Stratification

A new sepsis screening tool developed using machine learning was timelier and more discriminating than several benchmark screening tools, according to data published in the Annals of Emergency Medicine.

The new tool, the Risk of Sepsis (RoS) score, was developed using machine learning and compared with benchmark sepsis-screening tools such as the systemic inflammatory response syndrome, sequential organ failure assessment, quick sequential organ failure assessment, modified early warning score, and national early warning score. Investigators used retrospective electronic health record data from adult patients from 49 urban community hospital emergency departments over a 22-month period to derive and test the model.

A total of 2,759,529 records were obtained using the Rhee, et al¹ standard for clinical surveillance criteria as the definition of sepsis and the primary target for developing the model. The selection process consisted of 3 stages: (1) existing models for sepsis screening were reviewed, (2) consultation with local subject matter experts, and (3) supervised machine learning called gradient boosting. The performance metrics used were alert rate, area under the receiver operating characteristic curve, sensitivity, specificity, and precision.

Over the time periods assessed (1, 3, 6, 12, and 24 hours post index time) the RoS score was the most discriminant tool (area under the receiver operating characteristic curve 0.93 to 0.97). It was also more sensitive (67.7% vs 49.2% at 1 hour and 84.6% vs 80.4% at 24 hours) and precise (27.6% vs 12.2% at 1 hour and 28.8% vs 11.4% at 24 hours) compared with the next most discriminant tool, the sequential organ failure assessment. Overall, the RoS score performed significantly better across multiple dimensions of prognostic ability than the benchmarks.

Investigators acknowledged that the current lack of criterion standard diagnostics for sepsis is a study limitation, but they believe their approach using well-documented clinical criteria for sepsis as the model’s target was the best available option. They also noted that the key evaluation metrics were technical performance measures and “we did not address how this type of screening tool could be integrated into clinical practice,” and that further study would be required to “operationalize” the RoS in clinical practice. The RoS score may also suffer from a high false positive rate similar to other screening tools an area that also warrants further investigation.

According to investigators, the RoS score may be used in place of existing tools and RoS-driven screening is more sensitive than current assessments and may reduce the numbers of false alerts. However, they also cautioned that “despite our large cohort of patients, the demonstrable heterogeneity across our hospitals, and our robust training and testing framework, further external and independent evaluations are needed.”

References

1. Rhee C, Dantes RB, Epstein L, Klompas M. Using objective clinical data to track progress on preventing and treating sepsis: CDC’s new ‘Adult Sepsis Event’ surveillance strategy [published online September 25, 2018]. BMJ Qual Saf. doi:10.1136/bmjqs-2018-008331
2. Delahanty RJ, Alvarez J, Flynn LM, Sherwin RL, Jones SS. Development and evaluation of a machine learning model for the early identification of patients at risk for sepsis [published online January 17 2019]. Ann Emerg Med. doi:10.1016/j.annemergmed.2018.11.036

This article originally appeared on Infectious Disease Advisor