AI doctors fail by playing it too safe.

A new benchmark reveals that today’s best artificial intelligence models only look safe in the clinic because they constantly cry wolf.

If an AI doctor sends every patient with a mild headache to the emergency room, is it actually safe? On paper, yes, because it never misses a brain bleed. In reality, this behavior would instantly collapse a hospital triage system.

This tension is the core finding of MedSafe-Dx, a new safety-focused benchmark evaluating 12 frontier large language models on clinical diagnostic decisions. The results challenge the industry’s obsession with raw safety metrics. By showing that high safety scores often come from useless over-escalation, the benchmark exposes a critical flaw in how we measure clinical AI.

The safety trade-off

The benchmark tested models using a filtered 250-case subset of the DDx Plus dataset. Researchers measured two main things: Triage Success Rate (TSR), which penalizes unnecessary panic, and Safety Pass Rate (SPR), which tracks hard failures like missed emergencies. The results show a sharp conflict between the two.

• GPT-5 Chat achieved the highest TSR at 72.4%.
• Llama 4 Maverick and Grok 4.20 tied for second with a TSR of 71.2%.
• Llama 4 Maverick, an open-weight model, reached a near-perfect SPR of 96.8%.
• Reasoning-heavy models like o3-pro and DeepSeek R1 showed no systematic advantage in triage success.

This performance gap highlights a broader issue in clinical AI. As noted in a comprehensive review of large language models in healthcare, raw accuracy does not equal clinical utility. When models inflate their safety scores by over-escalating routine cases, they become a liability rather than a tool for efficiency. This creates a massive hurdle for deployment, especially in high-stress environments like emergency medicine, where clinical governance frameworks must balance safety with resource limits, as discussed in emergency medicine LLM frameworks.

The illusion of safety

The data shows that models with the highest safety scores achieved them by systematically over-escalating. They chose to protect their scores by flagging routine issues as urgent. This inverse relationship between safety and efficiency means healthcare systems cannot yet trust these models to run autonomously.

Of course, this benchmark has its limits. It relies on a relatively small cohort of 250 cases from a single dataset. Real-world clinical environments are far messier than structured datasets, and the benchmark does not yet account for multi-modal inputs like medical imaging or patient speech.

Until AI can balance caution with clinical reality, its role will remain strictly advisory.

Read the full preprint on medRxiv.