New research shows that explaining how medical AI works makes untrained users trust wrong answers, while doctors remain unaffected.

Can explaining how an AI thinks actually make it more dangerous? Technology companies assume that transparency builds trust and safety. But when we give those explanations to patients instead of doctors, that trust turns into blind faith.

A new study published in Nature Medicine tested this dynamic. Researchers evaluated 623 lay people and 153 primary care physicians (PCPs) using a fairness-based AI model for dermatological diagnoses. The system used multimodal large language models (LLMs) to explain its reasoning to the users.

The safety illusion

This finding challenges the industry obsession with explainable AI. For years, developers assumed that showing the “why” behind an algorithm’s decision would help users spot errors. Instead, it created an automation bias loop. When the AI was wrong, its polished explanation convinced laypeople to agree with the mistake.

PCPs, however, were resilient. Their medical training allowed them to benefit from the AI’s input without falling for its errors. This divergence suggests that medical AI must be tailored to the user’s specific expertise level, a concept supported by a review on human factors in JMIR Human Factors.

The study also revealed that timing is everything. Showing the AI’s diagnosis before a human could evaluate the case triggered a strong anchoring bias. This suggests that the current design of many clinical workflows, where AI flags cases upfront, might be fundamentally flawed.

Key trial findings

• The fairness-constrained model improved diagnostic accuracy and reduced skin-tone disparities for both groups.
• Lay users suffered from automation bias, showing higher accuracy when the AI was correct but dropping in accuracy when the AI erred.
• Experienced PCPs remained resilient to AI errors, benefiting from the tool regardless of whether the AI’s specific diagnosis was correct.
• Presenting the AI diagnosis early induced anchoring bias, warping human judgment before it could form independently.

The double-edged sword

This means consumer-facing health apps cannot use the same AI interfaces as clinical tools. If an app explains a skin lesion diagnosis to a worried patient, that patient may ignore their own doubts and skip a doctor’s visit based on a beautifully written, yet incorrect, AI explanation.

We must rethink how we present algorithmic decisions. As outlined in an ACM Transactions on Computing for Healthcare review, explainable AI is not a universal good. Without strict guardrails on who sees these explanations and when, transparency will continue to cause harm.

Read the full study in Nature Medicine.