Ignoring correct AI advice hurts diagnostic accuracy more than trusting wrong recommendations, but adding visual annotations helps clinicians make better calls.
Hospitals deploy artificial intelligence to catch deadly brain bleeds faster. Yet human clinicians frequently override correct algorithmic predictions, leading to missed diagnoses and worse overall performance than the AI operating alone.
A randomized crossover trial published in The Lancet Digital Health reveals that under-reliance on diagnostic software poses a greater threat to patient outcomes than blind trust. When doctors dismiss correct automated advice, clinical accuracy drops sharply. Visual explanations offer a potential bridge, helping physicians recognize when an algorithm is right without wasting precious time in emergency workflows.
The Cost of Disbelief
The trial evaluated 50 expert radiologists and 75 novice clinicians, including non-radiologist physicians and medical trainees. Between February and August 2023, each participant reviewed 50 head CT scans searching for intracranial hemorrhage under three distinct conditions: no AI support, basic AI predictions without annotations, and explainable AI featuring annotated scan overlays. To measure realistic clinical conditions, the AI provided accurate advice in 80% of cases and incorrect advice in 20%.
Both expert and novice clinicians struggled with appropriate reliance. However, under-reliance—defined as making a final decision that conflicted with accurate AI guidance—was associated with high uncertainty and caused a larger net decline in diagnostic performance than over-reliance on flawed predictions. Clinicians frequently second-guessed correct software outputs on tricky cases, missing subtle hemorrhages they should have caught.
Providing visual annotations mitigated this hesitance. Compared to unassisted review, accurate explainable AI delivered measurable improvements across key metrics:
- Diagnostic accuracy increased significantly (OR = 1.84, 95% CI 1.49–2.28, p < 0.0001), especially on complex hemorrhage cases.
- Diagnostic confidence rose among readers (b = 0.15, p < 0.0001, 95% CI 0.09–0.22).
- Explainable outputs were rated as noticeably more useful than basic predictions (b = 0.12, p = 0.0029, 0.04–0.20), particularly by novice medical staff.
Workflow Realities and Clinical Limits
The findings challenge the assumption that diagnostic software automatically streamlines hospital operations. Correct explainable AI failed to produce a statistically significant reduction in scan reading time compared to unassisted review (b = -1.81 seconds, p = 0.071, -3.78 to 0.16). While annotations helped doctors spot abnormalities accurately, verifying those features required active cognitive processing, neutralizing potential time savings.
This operational dynamic underscores ongoing challenges across diagnostic imaging and clinical decision support. Software overlays improve accuracy, but they do not automatically accelerate physician throughput or eliminate human error.
Crucially, the experiment took place in a controlled reader study rather than an active emergency department. While the simulated environment isolated human-AI interaction patterns, real-world clinical practice involves interruption, cognitive fatigue, and legal liabilities that could exacerbate both under-reliance and over-reliance. Current frameworks for explainable artificial intelligence in medical imaging must extend beyond visual annotations to guide appropriate human trust under pressure.
For health system leaders, the immediate takeaway is clear: software accuracy alone will not guarantee safer care. Training clinicians to properly calibrate their trust in AI tools is just as critical as the underlying model performance.
Read the full peer-reviewed study in The Lancet Digital Health.
