A machine learning tool accurately spots cognitive risk in sleep apnea patients, but faulty probability scoring proves why strong discrimination alone cannot justify clinical adoption.
A high area under the curve is the standard metric used to sell clinical AI models to hospitals. This study exposes why that metric is misleading. While the model achieved impressive discrimination across different cities, its raw risk estimates consistently overestimated patient risk in new settings, showing how a tool can rank patients correctly while providing faulty probabilities.
Strong discrimination across cities
Researchers built a random forest model to prioritize patients with obstructive sleep apnea (OSA) for formal cognitive assessments. The development cohort from Changchun included a 280-patient training set and a 120-patient internal hold-out. An external validation cohort from Shenzhen added 192 patients to test real-world transferability.
The prevalence of mild cognitive impairment (MCI) was consistent across groups, measuring 33.2% in the training set, 31.7% in the internal hold-out, and 29.7% in the external cohort. Nine features were selected for the final model: coffee consumption, physical activity, apnea-hypopnea index (AHI), minimum oxygen saturation (MSaO2), PHQ-9, GAD-7, Epworth Sleepiness Scale (ESS), CPSS-14, and PSQI total scores.
In the Shenzhen external validation cohort, the model achieved a 0.957 ROC-AUC, a 0.809 PR-AUC, and a low 0.051 Brier score. Similar efforts to map cognitive decline in sleep disorders, such as studies on the Multivariable Apnoea Prediction Index or degree centrality in severe OSA, have also struggled to balance algorithmic accuracy with practical clinical workflows.
The hidden calibration breakdown
Despite impressive discrimination figures, the external validation revealed significant miscalibration. The model produced a calibration intercept of -0.616 and a slope of 0.701 in Shenzhen. These numbers signal systematic probability overestimation. High discrimination means the software reliably ranks high-risk patients above low-risk ones, but miscalibration means the actual risk percentage printed on the screen cannot be trusted without local adjustment.
That distinction is the real story.
If clinicians use raw model outputs to triage care, overestimation can lead to unnecessary follow-up testing, increased workload, and patient anxiety. Algorithms designed to streamline care can easily backfire by artificially inflating clinical demand.
Key performance metrics
- Training set MCI prevalence: 33.2% (93 of 280 patients)
- External hold-out MCI prevalence: 29.7% (57 of 192 patients)
- External ROC-AUC: 0.957, showing high ranking accuracy
- External PR-AUC: 0.809, evaluating positive class precision
- Calibration metrics: Intercept of -0.616 and slope of 0.701, proving severe overestimation
Major limitations and takeaway
The model’s evidence base carries a severe demographic bias. Performance data comes almost entirely from male patients, meaning validity in women is completely unestablished. The software should not be used to evaluate female patients. Furthermore, feature selection was performed once on the full training set rather than nested within cross-validation folds, which introduces optimistic bias into early development metrics.
This software is a research screening tool, not a standalone diagnostic instrument. Health systems evaluating predictive models, including recent MCI diagnostic models for younger OSA cohorts, must mandate local recalibration assessments before deploying tools into daily triage workflows.
Read the full paper in BMC Psychiatry.
