A new study reveals that even the smartest medical AI models cannot accurately judge when they are wrong.
Would you trust a colleague who is absolutely certain of a diagnosis, yet wrong more than ten percent of the time? In clinical practice, knowing the limits of your own knowledge is a matter of life and death. Yet, the tech industry remains obsessed with raw accuracy scores, ignoring whether these systems actually know when to doubt themselves.
This disconnect is the real story. We are building highly capable machines that lack the basic self-awareness required for safe clinical triage.
A new study evaluated how well six major large language models—GPT-5, GPT-5-mini, GPT-5-nano, GPT-4o, Claude Sonnet 4.5, and Gemini 2.5—could estimate their own accuracy. Researchers put the models through 12,000 medical multiple-choice questions from the MedMCQA dataset. They measured the Expected Calibration Error (ECE), which is the mathematical gap between how confident an AI says it is and how often it actually gets the answer right.
The illusion of certainty
The data shows that high accuracy does not guarantee reliable uncertainty estimation. While larger models generally exhibit improved self-knowledge, uneven overconfidence remains a systemic issue across the board.
- Claude Sonnet 4.5 proved to be the most reliable, achieving the lowest calibration error of 0.06.
- GPT-4o sat at the bottom of the pack, registering the worst calibration error of 0.127.
- Clinical subjects mattered heavily, with “Skin” topics showing the best calibration at 0.041.
- “Social & Preventive Medicine” proved the hardest for models to judge, spiking to a worst-case calibration error of 0.141.
This mismatch is not unique to general medical exams. It mirrors findings in other highly specialized fields. For instance, recent research on cardiology knowledge alignment shows that general-purpose models frequently struggle to match their confidence with clinical reality when faced with complex, domain-specific questions.
Why pooled metrics fail
The real danger for healthcare providers lies in how we evaluate these systems. Looking at average performance across an entire model paints a dangerously misleading picture for clinical safety.
The data shows a threefold increase in calibration error between the best-performing and worst-performing medical specialties. If a hospital deploys an AI assistant based on its stellar average performance, clinicians will unknowingly face highly confident, incorrect advice when they cross into poorly calibrated specialties like preventive medicine. This is why we must stop treating AI as a uniform entity and begin mandating specialty-specific calibration reporting.
Of course, this study has limitations. Multiple-choice questions do not reflect the open-ended, messy nature of real-world patient consultations, where calibration errors could be even more unpredictable.
Read the full study in the Journal of Medical Systems.