A new benchmark reveals that how we format electronic health records for large language models completely alters their clinical accuracy.

If you feed a patient’s medical history to a top-tier AI, you expect the same clinical judgment regardless of how the text is spaced. A new benchmark shows this assumption is dangerously wrong. The way developers package Fast Healthcare Interoperability Resources (FHIR) data changes how AI “thinks.”

This challenges the industry assumption that LLMs are objective clinical reasoning engines. They are highly sensitive to data formatting.

The benchmarking trial

The study, FHIRBench, tested **four** frontier models: Claude Sonnet 4.5, GPT-5.4, DeepSeek V3.2, and Qwen3 32B. Researchers ran these models through **three** clinical tasks using **100** synthetic FHIR R4 patient bundles. They performed **7,200** evaluations per layer across two distinct testing methods: token-level F1 matching and a clinical rubric judged by an LLM.

The results show that simple formatting choices can break even the most advanced models.

Shocking performance gaps

The benchmark exposed deep flaws in how we evaluate clinical AI.

• Formatting choice changes everything: Condensed formats beat Raw JSON on token matching for **three out of four** models (Wilcoxon p < 10^-17), but Raw JSON scored higher on actual clinical quality (p < 10^-7).
• Narrative formats are highly efficient: They achieved **95%** of Raw JSON’s clinical quality while using **83%** fewer tokens.
• Model rankings flipped: Claude ranked last on token matching but first on clinical quality (p = 1.0 x 10^-6).
• A major safety gap emerged: Llama 3.1 70B suffered a **100%** inference failure rate on complex patients, even though the data fit inside its context window.

The safety hazard

This is not just a technical quirk. It is a patient safety hazard.

If a model like Llama 3.1 70B completely blanks out when handed a complex patient file, the system is useless for the very people who need it most. The industry must stop assuming that a large context window means a model can actually process complex data.

Furthermore, the ranking reversal proves that simple token-matching metrics are useless for measuring clinical safety. A model can fail to copy words exactly but still make the correct diagnosis. We cannot rely on a single “best” data format either. The study found a significant interaction between models and serializers (Friedman p = 0.0009). GPT-5.4 thrives on Raw JSON, while open-weight models perform better with compressed formats.

The study has limitations. It relied on synthetic patient data, which may not capture the messy realities of real-world hospital records. Still, the implications are clear. Healthcare tech companies cannot just plug an LLM into an existing database. They must build custom middleware to translate data specifically for their chosen model.

Read the full study in medRxiv.