A machine learning model accurately ranked miscarriage risk for pregnant patients with PCOS, but sharp performance drops mean it cannot guide treatment decisions.
High hopes, declining performance
Confirming a clinical pregnancy after fresh IVF or ICSI brings immense relief to women with polycystic ovary syndrome (PCOS). Yet these patients face an elevated risk of early pregnancy loss. Clinics lack reliable tools to flag high-risk individuals before outcomes occur.
To address this gap, researchers developed a machine learning model to stratify miscarriage risk using routine clinical data. They trained an XGBoost algorithm on 7,241 cycles with 973 miscarriages (a 13.4% event rate). During internal validation, the tool performed exceptionally well, achieving an area under the receiver operating characteristic curve (AUC) of 0.885. This work builds on prior efforts to map assisted reproductive outcomes in PCOS patients, such as models analyzing how an early LH surge jeopardizes ART outcomes.
However, performance degraded when tested outside the development environment.
In geographic external validation (1,170 cycles, 146 miscarriages), discrimination fell to an AUC of 0.787 with a Brier score of 0.099. When tested in a prospective temporal cohort of 500 cycles (62 miscarriages), accuracy dropped further to an AUC of 0.763 with a Brier score of 0.104.
That attenuation is the central problem.
What the algorithm sees
High internal metrics often create false confidence. When algorithms move across geographic boundaries or timeframes, underlying population shifts trigger calibration drift and performance loss.
In this trial, raw XGBoost predictions severely overestimated absolute danger, predicting a mean risk of 30.3% against an actual observed rate of 13.4%. Despite calibration issues, Shapley additive explanations identified clear biological signals. Anti-Müllerian hormone, basal luteinizing hormone, and endometrial thickness on the day of human chorionic gonadotropin administration emerged as top predictors. These features align with broad initiatives using artificial intelligence to predict reproductive outcomes in PCOS.
While the model correctly enriched event detection across risk quartiles, its positive predictive value remained modest.
Clinical key takeaways
- Risk ranking only: The tool reliably ranks relative risk but cannot diagnose or select specific drug interventions.
- Mandatory re-calibration: Clinics cannot deploy this model out of the box; local calibration monitoring is essential.
- No clinical substitute: Discrimination loss means clinicians must view outputs as supplementary counseling aids.
This study demonstrates that predicting pregnancy loss requires more than biological variables captured at embryo transfer. Until models maintain absolute calibration across diverse populations, algorithms should only aid patient counseling rather than drive protocol changes.
Read the full paper in the Journal of Medical Internet Research.
