⚡ Quick Summary
This study compared the performance of artificial intelligence (AI) early warning scores with traditional non-AI scores in identifying clinical deterioration among hospitalized patients. The eCART score demonstrated superior accuracy, highlighting the potential of AI in enhancing patient monitoring systems.
🔍 Key Details
- 📊 Dataset: 362,926 patient encounters across 7 hospitals
- 🧩 Scores compared: eCART, NEWS2, NEWS, Rothman Index, EDI, MEWS
- ⚙️ Study design: Retrospective cohort study
- 🏆 Main outcome: Clinical deterioration events (ICU transfer or death within 24 hours)
🔑 Key Takeaways
- 📈 eCART achieved the highest area under the curve (AUC) at 0.895.
- 💡 NEWS2 and NEWS also performed well, with AUCs of 0.831 and 0.829, respectively.
- 🔍 EDI and MEWS had lower AUCs of 0.808 and 0.757.
- ⏳ Moderate-risk thresholds provided a median lead time of at least 20 hours for all scores.
- ⚠️ Positive predictive values (PPVs) varied significantly, with eCART showing a high of 17.3% at moderate risk.
- 🌟 NEWS outperformed EDI, indicating the effectiveness of non-AI tools.
- 🔍 Transparency and oversight of early warning tools may be necessary due to performance variability.
📚 Background
Early warning scores are critical tools in hospitals for identifying patients at risk of clinical deterioration. These scores help healthcare providers make timely interventions, potentially saving lives. However, the comparative effectiveness of AI-based scores versus traditional methods has been underexplored, prompting this study to fill that gap.
🗒️ Study
Conducted within the Yale New Haven Health System, this retrospective cohort study analyzed data from adult medical-surgical ward encounters between March 2019 and November 2023. The researchers aimed to evaluate the performance of three AI-based early warning scores against three traditional, publicly available scores.
📈 Results
The study revealed that among the 362,926 patient encounters, 16,693 patients (4.6%) experienced clinical deterioration. The eCART score outperformed all other scores, achieving an AUC of 0.895. In terms of lead time, eCART provided a median of 11 hours at high-risk thresholds, allowing for timely clinical interventions.
🌍 Impact and Implications
The findings from this study underscore the potential of AI in enhancing patient monitoring systems. By demonstrating that eCART can identify deteriorating patients more accurately and with fewer false alarms, this research paves the way for integrating AI technologies into clinical practice. Improved early warning systems could lead to better patient outcomes and more efficient use of healthcare resources.
🔮 Conclusion
This study highlights the significant advantages of AI-based early warning scores, particularly eCART, in identifying clinical deterioration. As healthcare continues to evolve, the integration of AI tools could revolutionize patient monitoring, ensuring timely interventions and improved outcomes. Further research and transparency in the development of these tools will be essential for their successful implementation in clinical settings.
💬 Your comments
What are your thoughts on the use of AI in early warning systems? Do you believe it can significantly improve patient care? 💬 Share your insights in the comments below or connect with us on social media:
Early Warning Scores With and Without Artificial Intelligence.
Abstract
IMPORTANCE: Early warning decision support tools to identify clinical deterioration in the hospital are widely used, but there is little information on their comparative performance.
OBJECTIVE: To compare 3 proprietary artificial intelligence (AI) early warning scores and 3 publicly available simple aggregated weighted scores.
DESIGN, SETTING, AND PARTICIPANTS: This retrospective cohort study was performed at 7 hospitals in the Yale New Haven Health System. All consecutive adult medical-surgical ward hospital encounters between March 9, 2019, and November 9, 2023, were included.
EXPOSURES: Simultaneous Epic Deterioration Index (EDI), Rothman Index (RI), eCARTv5 (eCART), Modified Early Warning Score (MEWS), National Early Warning Score (NEWS), and NEWS2 scores.
MAIN OUTCOMES AND MEASURES: Clinical deterioration, defined as a transfer from ward to intensive care unit or death within 24 hours of an observation.
RESULTS: Of the 362 926 patient encounters (median patient age, 64 [IQR, 47-77] years; 200 642 [55.3%] female), 16 693 (4.6%) experienced a clinical deterioration event. eCART had the highest area under the receiver operating characteristic curve at 0.895 (95% CI, 0.891-0.900), followed by NEWS2 at 0.831 (95% CI, 0.826-0.836), NEWS at 0.829 (95% CI, 0.824-0.835), RI at 0.828 (95% CI, 0.823-0.834), EDI at 0.808 (95% CI, 0.802-0.812), and MEWS at 0.757 (95% CI, 0.750-0.764). After matching scores at the moderate-risk sensitivity level for a NEWS score of 5, overall positive predictive values (PPVs) ranged from a low of 6.3% (95% CI, 6.1%-6.4%) for an EDI score of 41 to a high of 17.3% (95% CI, 16.9%-17.8%) for an eCART score of 94. Matching scores at the high-risk specificity of a NEWS score of 7 yielded overall PPVs ranging from a low of 14.5% (95% CI, 14.0%-15.2%) for an EDI score of 54 to a high of 23.3% (95% CI, 22.7%-24.2%) for an eCART score of 97. The moderate-risk thresholds provided a median of at least 20 hours of lead time for all the scores. Median lead time at the high-risk threshold was 11 (IQR, 0-69) hours for eCART, 8 (IQR, 0-63) hours for NEWS, 6 (IQR, 0-62) hours for NEWS2, 5 (IQR, 0-56) hours for MEWS, 1 (IQR, 0-39) hour for EDI, and 0 (IQR, 0-42) hours for RI.
CONCLUSIONS AND RELEVANCE: In this cohort study of inpatient encounters, eCART outperformed the other AI and non-AI scores, identifying more deteriorating patients with fewer false alarms and sufficient time to intervene. NEWS, a non-AI, publicly available early warning score, significantly outperformed EDI. Given the wide variation in accuracy, additional transparency and oversight of early warning tools may be warranted.
Author: [‘Edelson DP’, ‘Churpek MM’, ‘Carey KA’, ‘Lin Z’, ‘Huang C’, ‘Siner JM’, ‘Johnson J’, ‘Krumholz HM’, ‘Rhodes DJ’]
Journal: JAMA Netw Open
Citation: Edelson DP, et al. Early Warning Scores With and Without Artificial Intelligence. Early Warning Scores With and Without Artificial Intelligence. 2024; 7:e2438986. doi: 10.1001/jamanetworkopen.2024.38986