A multimodal machine learning model for delirium risk maintained strong accuracy when deployed live at a separate hospital without retraining, though high false-positive rates threaten to burden nursing staff.
Predictive Power Across Sites
Most clinical algorithms degrade when moved to a new hospital. Patient demographics shift, charting habits differ, and underlying care patterns change. This trial challenged that trend by taking a multimodal fusion model trained on natural language processing notes from Mount Sinai Hospital (MSH) and running it live at Mount Sinai Morningside Hospital (MSM).
The two populations differed significantly. Patients at MSM were older, with a median age of 75.79 years compared to 72.62 years at MSH. They also had distinct profiles across sex, race, ethnicity, surgical versus medical primary diagnoses, length of stay, and baseline dementia prevalence.
Despite these baseline differences across 33,095 total non-ICU admissions, the model performed remarkably well across both sites. It achieved an area under the receiver operating characteristic curve (AUROC) of 0.91 at the external MSM site, compared to 0.94 at the original MSH development site.
That stability matters. It proves that predictive models using unstructured clinical text can maintain discrimination across separate facilities without requiring local fine-tuning.
Higher Yields, Heavier Workload
Accuracy on paper means little if clinical detection stays stagnant. After deploying the model live across 9,020 MSM admissions, the mean monthly delirium detection rate jumped from 9.23% to 19.59%.
Catching twice as many cases of delirium opens the door to early intervention and better patient management. But the operational cost is high. The model achieved a positive predictive value of just 25.9% at MSM and 19.4% at MSH.
That low precision means roughly three out of every four flagged patients did not actually have delirium.
When software generates that many false alarms, clinicians suffer from alert fatigue. Instead of replacing clinical judgment, the tool forces staff to spend time evaluating false positives, which can undermine trust in automated alerts.
The Operational Reality
Diagnostic accuracy is not the same as improved patient outcomes. While the model succeeded at identifying at-risk patients, this quality improvement study did not demonstrate reduced stay durations or lower complication rates.
The study cohort was also limited to non-ICU patients aged 60 and older within a single health system. The pre-deployment cohort at MSM (median age 78.00 years) was significantly older than the post-deployment group (median age 75.79 years), introducing potential historical bias.
Hospitals considering similar AI tools should focus on how alerts trigger care pathways. High discriminative performance proves an algorithm works, but without strict workflows to filter out false alarms, the clinical workload may outweigh the diagnostic gain.
Read the full study in JAMA Network Open.
