AI algorithms built to confirm whether chemotherapy has eradicated esophageal tumors are not reliable enough to guide surgical decisions.
Surgeons face a high-stakes choice after neoadjuvant therapy for esophageal cancer. If an imaging tool correctly identifies a pathological complete response, a patient might safely avoid highly invasive esophagectomy. But if the tool misses residual disease, delaying surgery allows lethal tumors to progress.
A systematic review in the Journal of Medical Internet Research demonstrates that current algorithms carry too much clinical uncertainty to justify changing surgical plans. While average performance looks acceptable on paper, performance swings wildly when algorithms move to new medical centers.
The gap between internal training data and external reality is the core problem.
Across 33 evaluated studies, algorithms performed reasonably well within their original training cohorts. But when tested on independent patient groups, diagnostic reliability dropped, revealing wide prediction intervals that undermine clinical trust.
Performance Drops in External Validation
The meta-analysis evaluated 2,668 validation observations across internal and external testing groups. Key statistical findings include:
- Internal Validation (27 datasets, 1,891 patients): Achieved an average area under the curve (AUC) of 0.86, with 0.77 sensitivity and 0.82 specificity.
- External Validation (16 datasets, 777 patients): Average AUC dropped to 0.84, with 0.73 sensitivity and 0.83 specificity.
- Extreme Performance Swings: The 95% prediction interval for external sensitivity ranged from 0.34 to 0.93, while specificity ranged from 0.36 to 0.98.
That 0.34 lower bound for external sensitivity means an algorithm could miss residual cancer in two out of three patients when deployed at a new site. Relying on such erratic predictions puts patients at direct risk of unmanaged tumor growth.
Geographic Isolation and Bias
These findings mirror broader challenges in computer-assisted oncologic staging. Retrospective deep learning assessments, such as nationwide endoscopic studies in Japan, also highlight the need for rigorous multicenter testing before software can inform surgical decision-making. Similar validation hurdles exist when evaluating endoscopic response to chemoradiotherapy across varying treatment regimens.
The dataset behind this meta-analysis suffers from severe geographic concentration. Out of 33 analyzed studies, 32 were conducted exclusively in China, severely limiting how well these findings apply to Western patient populations or different scanner types.
Furthermore, 12 studies carried a high risk of validation bias under PROBAST+AI criteria. The GRADE certainty for all primary diagnostic outcomes was rated as very low.
Practical Takeaway for Clinicians
Imaging-based AI cannot replace post-neoadjuvant pathological workups or justify organ preservation strategies today. Health systems should treat these algorithms as experimental decision support only, requiring prospective trials and clear calibration thresholds before considering them for active surveillance protocols.
Data source: Journal of Medical Internet Research (2026)
