Using large language models to clean up brain-computer interface outputs introduces a dangerous new failure mode: fluent, highly confident lies that alter what paralyzed patients are actually trying to say.

When a paralyzed patient uses a brain-computer interface (BCI) to speak, a typo is annoying. But what happens when an AI helper rewrites that typo into an entirely different, perfectly polished sentence?

This is “intent drift.” It challenges the current rush to slap large language models (LLMs) onto medical hardware as a quick fix for noisy sensors. Instead of just correcting typos, the AI silently hijacks the message. This means we cannot treat LLMs as neutral translation layers. If the machine decides what the patient “meant” to say, it strips away the user’s autonomy.

The high cost of fluency

A recent in-silico benchmark tested **20 open-weight models** processing over **4.2 million generations** of text corrupted by simulated typing errors. When character error rates rose from 0% to 40%, unintended semantic drift in ALS message-banking vocabulary skyrocketed from **2.2% to 60.3%**. The odds of the AI hallucinating a new meaning increased by **2.30 times** for every 10-percentage-point rise in input corruption. Message-critical content was especially vulnerable, carrying a small excess risk of drift with a rule-free odds ratio of **1.10**.

The real danger is that these models lie with a straight face. While the AI’s stated confidence scored an AUROC of **0.83** for detecting faithful outputs, it was poorly calibrated, yielding an expected calibration error of **0.32**. In fact, **28.4%** of the outputs generated with a confidence score of 90% or higher were completely unfaithful to the original message. The AI is highly confident even when it is completely wrong.

No easy policy fix

Engineers cannot simply program their way out of this with clever interface rules. No tested policy eliminated the drift. Conservative editing and abstention lowered the rate, but the best-performing policy still drifted on **18.0 per 100** messages. Furthermore, the ratio of helpful rescues to fluent errors fell below 1 once input corruption hit **20% to 30%**.

Even pooling models together offers little relief. A 2,281-item panel yielded moderate agreement between the AI ensemble and human consensus, with a kappa of **0.41**. Relying on this ensemble correction only lowered pooled drift from **31.4% to 28.3%**.

Key benchmark results

• Intent drift spiked to 60.3% under heavy sensor noise.
• Nearly 30% of high-confidence AI corrections were actually incorrect.
• The best interface policy still drifted on 18.0 per 100 messages.

We must rethink the safety profile of these systems. This study was simulated in-silico, meaning we still need prospective, human-in-the-loop trials to see how real patients react when their words are twisted. But the takeaway is clear. When we prioritize smooth, fluent sentences over raw, messy accuracy, we risk taking away the patient’s actual voice.

Read the full study in medRxiv.