Prostate cancer AI models can match expert diagnostic accuracy across international populations, but performance collapses depending on the hospital scan quality.
Algorithms for detecting prostate lesions on MRI often fail when moved from pristine clinical trials to everyday hospitals. The SCARLET-1 study tested whether a single AI system could maintain diagnostic accuracy across diverse real-world diagnostic and screening cohorts. The results show the software meets criteria for diagnostic interchangeability with human radiologists, but severe performance gaps across different sites reveal a hidden operational risk.
Global Accuracy Meets Industry Standards
To train and internally test the algorithm, researchers gathered 20,471 prostate MRI examinations from 19,278 patients spanning 26 cities and 14 countries. The system was then evaluated on an external validation cohort of 1,974 patients across 20 cities in 12 countries. This broad evaluation builds on early work in automated lesion detection on biparametric MRI to see if models hold up globally.
The primary target was identifying clinically significant cancer, defined as Gleason grade group 2 or higher. The model matched human diagnostic readings across three major external screening and diagnostic datasets:
- PRIME diagnostic dataset: Diagnostic agreement reached 83.0% (395 of 476) with an AUC of 0.86.
- STHLM3-MRI screening dataset: Diagnostic agreement reached 87.9% (974 of 1108) with an AUC of 0.89.
- IP1-PROSTAGRAM screening dataset: Diagnostic agreement reached 87.7% (342 of 390) with an AUC of 0.84.
These numbers met the study’s prespecified threshold for interchangeability with average radiologist performance. Demographic factors like patient age, ethnicity, cancer location, and baseline risk caused minimal variation, shifting the mean AUC by just 0.02.
Image Quality Controls Real-World Success
That high overall accuracy hides significant site-level volatility. While demographic transfers were smooth, localized hardware and technical execution heavily altered the tool’s performance.
Across 25 clinical sites, accuracy swung dramatically. Fully 28% of sites saw the tool perform poorly with an AUC below 0.80. Conversely, 32% of sites achieved an AUC of 0.90 or higher. Image quality was the decisive factor: AUC scores dropped from 0.88 on optimal scans down to 0.78 on degraded images.
This volatility mirrors broader concerns detailed in deep learning prostate MRI reviews regarding model generalization across different scanner vendors. AI cannot compensate for poor radiology technique at the local level.
Why This Matters
Health systems cannot deploy prostate AI out of the box and expect uniform quality. Because a site’s baseline scan clarity directly governs whether the algorithm performs like an expert or underperforms, clinics must standardize image acquisition protocols before trusting software outputs.
Retrospective agreement with past readings does not prove AI reduces unnecessary biopsies or speeds up patient care. Until prospective trials measure clinical workflow impacts, healthcare providers should view AI as a quality-control backup rather than a replacement for human readers.
Read the full paper in Radiology.
