⚡ Quick Summary
This study evaluated the effectiveness of a clinical large language model (LLM), Gatortron, in predicting antimicrobial resistance (AMR) in hospital-onset sepsis compared to traditional machine learning methods. The LLM demonstrated superior performance, achieving an AUC of 0.73 for MRSA prediction, highlighting its potential for enhancing clinical decision-making in critical care settings.
🔍 Key Details
- 📊 Dataset: EHR data from approximately 150,000 hospitalizations with bacterial infections
- 🧩 Features used: Simplified electronic health record (EHR) data
- ⚙️ Technology: Gatortron (LLM) vs. traditional machine learning models
- 🏆 Performance: LLM AUC 0.73 vs. ML AUC 0.66 for MRSA prediction
🔑 Key Takeaways
- 📊 Antimicrobial resistance is a critical challenge in managing hospital-onset sepsis.
- 💡 Gatortron outperformed traditional machine learning models in predicting MRSA.
- 👩🔬 Study involved 2,019 hospital-onset sepsis encounters, with 45% identified as AMR pathogens.
- 🏆 LLM achieved an AUC of 0.73 and an F1 score of 0.43.
- 🤖 Traditional ML models had an AUC of 0.66 and an F1 score of 0.16.
- 📈 Negative predictive value for MRSA prediction using LLM was at least 90% across most infection presentations.
- 🌍 Study conducted at a large tertiary care healthcare system from 2010 to 2023.
- 🔍 Further refinement of LLMs is needed to enhance sensitivity and clinical applicability.
📚 Background
The rise of antimicrobial resistance (AMR) poses a significant threat to patient outcomes, particularly in critically ill populations. Effective empiric antimicrobial therapy is essential for managing infections, especially in cases of hospital-onset sepsis where AMR is prevalent. Traditional methods of predicting AMR often lack scalability and generalizability, necessitating innovative approaches such as the use of artificial intelligence and large language models (LLMs).
🗒️ Study
This study aimed to compare the predictive capabilities of Gatortron, a publicly available clinical LLM, against traditional machine learning models in forecasting AMR and MRSA-specific patterns in a cohort of hospital-onset sepsis patients. Researchers analyzed EHR data from a substantial number of hospitalizations to assess the models’ performance in real-world clinical settings.
📈 Results
The results indicated that the LLM significantly outperformed traditional machine learning models in predicting MRSA, achieving an AUC of 0.73 compared to 0.66 for the best traditional model. Additionally, the LLM’s F1 score of 0.43 was markedly higher than the 0.16 achieved by traditional models. The negative predictive value for MRSA prediction using the LLM was consistently above 90%, underscoring its reliability in clinical applications.
🌍 Impact and Implications
The findings from this study suggest that leveraging LLMs like Gatortron can enhance the prediction of AMR in hospital settings, potentially leading to improved patient management and outcomes. By integrating EHR data with advanced AI technologies, healthcare providers can make more informed decisions regarding antimicrobial therapy, ultimately addressing the pressing challenge of AMR in critical care environments.
🔮 Conclusion
This study highlights the transformative potential of large language models in predicting antimicrobial resistance, showcasing their superior performance over traditional machine learning methods. As we continue to refine these technologies, the integration of AI in clinical practice could significantly improve the management of infections in critically ill patients. Ongoing research and development in this area are essential for enhancing the sensitivity and applicability of these models in real-world healthcare settings.
💬 Your comments
What are your thoughts on the use of AI and large language models in predicting antimicrobial resistance? We would love to hear your insights! 💬 Leave your comments below or connect with us on social media:
Comparative Evaluation of Clinical Large Language Models and Machine Learning to Predict Antimicrobial Resistance in Hospital-Onset Sepsis.
Abstract
Approaches to guide empiric antimicrobial therapy are needed, especially in critically ill populations with prevalent antimicrobial resistance (AMR). While artificial intelligence shows promise in predicting AMR, scalable and generalizable prediction models are essential for broad clinical adoption. We utilized a publicly available clinical large language model (LLM), Gatortron, in comparison to traditional machine learning, to predict AMR and methicillin-resistant Staphylococcus aureus (MRSA)-specific patterns within a hospital-onset sepsis cohort using electronic health record (EHR) data available at time of illness onset. EHR data from approximately 150,000 hospitalizations with a documented bacterial infection at a large tertiary care healthcare system between 2010 and 2023 were examined. Among 2,019 eligible hospital-onset sepsis encounters, an AMR pathogen was identified in 911 (45%) and MRSA was isolated in 234 (26%). LLMs outperformed traditional models in predicting MRSA, achieving an AUC of 0.73 compared to 0.66 for the best traditional ML model, with superior F1 scores (0.43 vs. 0.16 for ML). Negative predictive value for MRSA prediction using LLM was at least 90% across majority of infection presentations. The LLM’s superior prediction using a relatively simplified feature set demonstrates the potential of leveraging EHR data for early resistance prediction, though further refinement is needed to enhance sensitivity and clinical applicability.
Author: [‘Cohen SA’, ‘Chen Z’, ‘Bian J’, ‘Boucher C’, ‘Wu Y’, ‘Prosperi M’]
Journal: Artif Intell Med Conf Artif Intell Med (2005-)
Citation: Cohen SA, et al. Comparative Evaluation of Clinical Large Language Models and Machine Learning to Predict Antimicrobial Resistance in Hospital-Onset Sepsis. Comparative Evaluation of Clinical Large Language Models and Machine Learning to Predict Antimicrobial Resistance in Hospital-Onset Sepsis. 2025; 15734:65-76. doi: 10.1007/978-3-031-95838-0_7