A new study reveals that how researchers validate clinical prediction models matters far more than the complexity of the algorithms they build.

In Bangladesh, caesarean sections reached 51.8% of all deliveries in 2025, with elective procedures before labor accounting for 31.6%. To curb unnecessary surgeries, policy researchers are building machine learning models to audit where scheduled operations outpace clinical needs. But these models are typically validated in ways that flatter their real-world utility.

This practice challenges the industry’s obsession with algorithmic complexity. Engineers spend months tuning advanced neural networks or gradient boosting systems when simple models perform identically. The real failure point is geographic and temporal drift, which standard validation methods routinely ignore.

The geographic blind spot

Researchers trained four models—logistic regression, elastic net, random forest, and gradient boosting—on 9,538 women from the 2025 Bangladesh Multiple Indicator Cluster Survey. Under cluster-grouped validation, the area under the receiver operating characteristic curve (AUC) for all models sat in a tight window from 0.724 to 0.736. Simple logistic regression performed just as well as complex machine learning, with a maximum spread of just 0.012 between them.

The real drop in performance occurred when models faced unfamiliar territory. While grouping validation folds by sampling cluster barely changed discrimination by at most 0.0004, withholding an entire division caused the AUC to plunge by 0.044 to 0.060. This geographic penalty is over 100 times larger than the cluster adjustment, proving that predictability is highly localized.

Time degrades performance

The models also failed to travel through time. A model trained on 2019 data lost 0.083 in AUC when applied to 2025. The predictors themselves shifted, showing only a moderate Spearman rank correlation of 0.61 between the two periods.

• C-sections accounted for 51.8% of 2025 births, with elective surgeries at 31.6%.
• Model choice barely mattered, with AUCs clustered tightly between 0.724 and 0.736.
• Geographic holdouts reduced AUC by up to 0.060, showing high local sensitivity.
• Temporal transfer from 2019 to 2025 caused a steep AUC drop of 0.083.

On the positive side, calibration held steady across all wealth quintiles, both urban and rural residences, and seven of eight divisions. Sylhet was the sole exception where calibration failed.

Rethinking model validation

This analysis forces a rethink of how we evaluate health AI. If predictability is local, then standard cross-validation does not measure what a model would do in a district or year it has never seen. Geographic holdouts are the minimum standard for realistic testing.

This analysis is based on research published in medRxiv.