New research reveals that top-tier artificial intelligence models pass complex orthopedic exams not through clinical reasoning, but by exploiting test-pattern shortcuts.
Can an AI safely diagnose a bone fracture if it does not even look at the X-ray? A recent study evaluating large language models on the Orthopaedic In-Training Examination (OITE) suggests the answer is a troubling no. The models are passing, but they are doing it by gaming the multiple-choice format.
This challenges the assumption that high exam scores equal clinical competence. It exposes medical benchmarks as deeply flawed. We are measuring test-taking optimization, not clinical safety.
The test-taking illusion
Researchers tested eight models on 792 OITE questions from 2020 through 2024. This dataset included 434 questions with images and 358 without. The models spanned three open-source Ministral-3 versions (3B, 8B, and 14B) and five proprietary giants, including Claude Haiku-4.5, Sonnet-4.6, Opus-4.8, GPT-5.6 Luna, and GPT-5.6 Terra.
At first glance, the results look stellar. OpenAI’s GPT-5.6 Terra hit a peak accuracy of 94.13% on non-image questions and 81.80% on image-containing questions. But these high scores mask a hollow core.
Blind to the images
When researchers stripped the images away entirely, the models did not stumble. For questions that originally had images, pooled accuracy was 59.13% with complete information and remained exactly 59.13% without the images. Removing the visual data did not significantly hurt a single model.
Instead, the text-based clinical vignette did all the heavy lifting. The study mapped this reliance by systematically stripping information:
- Removing the clinical vignette dropped pooled accuracy to 49.05%.
- Removing both the vignette and images pushed accuracy down to 45.68%.
- Removing the vignette, images, and the question itself, leaving only the answer options, still yielded 36.49% accuracy.
- For non-image questions, removing the vignette dropped accuracy from 72.94% to 53.53%, and then to 37.36% with answer options alone.
Spurious shortcuts
The most damning finding is how models perform when given nothing but the multiple-choice options. With no question, no clinical context, and no images, every model still beat the 25% random guess threshold. Claude Opus 4.8 scored 45.62% on image-question options, and GPT-5.6 Terra hit 46.09% on non-image options.
This is not clinical reasoning. It is pattern matching. The models are exploiting linguistic cues and spurious correlations embedded in how medical test questions are written.
To make matters worse, their explanations sounded highly professional even when they were wrong. Using BioMedBERT to evaluate semantic similarity, researchers found that the quality of the AI’s explanation did not correlate with accuracy, showing a Spearman correlation of -0.17 (P=0.29). A model can write a highly convincing, medically sound essay while confidently picking the wrong treatment.
This matters because medical licensing bodies are increasingly using these exams to validate AI safety. If an AI can pass a bone surgery exam without looking at the bone, the benchmark is broken. We must stop treating exam scores as a proxy for clinical safety.
Read the full study in medRxiv.
