Large language models usually resist ordering low-value medical tests, but occasional lapses still risk driving unnecessary diagnostic cascades.
Healthcare systems lose billions annually to low-value medical care. When evaluating AI models against Choosing Wisely guidelines, top systems successfully avoided inappropriate medical orders in 95% of clinical scenarios.
The Ultrasound Trap
The remaining 5% of responses recommended potentially unnecessary care. Thyroid ultrasounds emerged as the primary source of inappropriate recommendations.
While a 5% error rate seems low, automated clinical guidance operates at scale. A single recurring misstep in routine imaging can trigger invasive follow-up biopsies, patient anxiety, and downstream costs. Diagnostic accuracy in simulated tests does not equal safe autonomous performance in active clinical workflows.
Operational Reality
These findings show AI models can reinforce evidence-based guidelines, but they cannot replace physician oversight. Health systems deploying AI for decision support must implement strict guardrails around imaging referrals to prevent automated care cascades.
