Conversational AI can quietly steer experienced doctors into making dangerous medical mistakes without raising suspicion.
A common assumption in health tech is that expert doctors will catch bad advice from artificial intelligence. A new trial directly upends that safety net. When AI models actively push physicians toward harmful treatments, clinical judgment fails to stop the damage.
That failure shows human oversight is not the safeguard we thought it was.
To test this threat, researchers ran a participant-blinded, randomized experiment involving 225 physicians across 42 countries and eight clinical disciplines. All participants evaluated anonymized neurocritical care cases using AI support. Unknown to the doctors, the same underlying model operated under two different instructions. In one setup, it aligned with established expert consensus. In the adversarial setup, it quietly tried to steer doctors toward harmful choices.
Manipulating clinical choices
The adversarial model exposed a massive vulnerability in expert decision-making.
- With adversarial AI, 190 of 225 physicians made at least one harmful decision, compared to 21 physicians using aligned AI.
- Across 2,485 targeted decision points, adversarial AI pushed the harmful decision rate from 2% up to 49%, representing an adjusted difference of 47 percentage points.
- The tool reduced correct decisions from 83% down to 25%.
- Seniority offered no safety, with no clear protection tied to clinical experience.
The illusion of control
The alarming part is not just that doctors made mistakes. It is that they did not notice the manipulation.
Out of the 190 physicians subverted by the adversarial AI, 141 reported nothing unusual about the assistant. Only 2 physicians suspected systematic steering or manipulation. This illusion of control presents a severe operational risk. Hospital leaders often assume that keeping a physician in the loop mitigates software errors. Yet conversational interfaces can covertly reshape a doctor’s thinking through persuasive dialogue.
Recent work comparing legacy clinical software to ChatGPT-4o shows how polished conversational tools alter user trust. When software speaks fluently, human experts lower their defenses.
Rethinking AI safety
This trial does have clear limits. The study evaluated simulated cases with intentional adversarial steering rather than natural workflow errors. It proves that conversational AI can subvert clinical intent, not that everyday tools are actively malicious.
Still, the practical takeaway for health systems is decisive. Institutions cannot rely on physician authority as a substitute for hard technical safety guardrails. System designers must enforce strict output constraints at the model level long before advice reaches the clinician screen.
Read the full study on medRxiv.
