Forcing local clinical AI models to follow strict formatting rules stops software crashes but does not guarantee safe medical editing.
Can we trust a local, budget-friendly AI to format clinical notes without crashing hospital software? Many health tech teams focus on clinical accuracy first. But if an AI output cannot be parsed by an electronic health record, its medical accuracy is useless. This evaluation reveals that while schema enforcement guarantees clean code, it masks a deeper problem: the underlying text editing remains highly unpredictable.
Researchers tested three open-weight models in the 7- to 8-billion-parameter range: Llama3-Med42-8B, Meta-Llama-3-8B-Instruct, and Mistral-7B-Instruct-v0.3. They ran 70 de-identified English admission notes, split evenly between 35 internal medicine and 35 surgical cases. Using consumer-grade hardware and LM Studio, they executed 1260 local inferences across three independent runs.
The formatting failure rate
Without strict enforcement, the models struggled with basic JSON formatting. Mistral-7B-Instruct-v0.3 performed the worst, managing only 74.3% to 78.6% first-pass validity per run. It left 18.6% to 21.4% persistent failures even after retries, failing completely on 17 of the 70 notes.
Prompt structure also caused unexpected friction. When researchers embedded system instructions directly into user messages for Meta-Llama-3-8B-Instruct, first-attempt failures jumped to 7.9% (55 out of 700). Keeping system and user roles separate kept failures down to 1.7% (12 out of 700).
Enforcement is not accuracy
Enforcing a schema solved the formatting crisis. Under this condition, every single model achieved 100% first-pass validity, delivering 70 out of 70 compliant outputs across all runs.
But perfect formatting is a mirage.
Even with flawless JSON structures, the models differed wildly in their actual editing. They showed significant variation in verbosity, how they preserved numeric tokens, and how they handled uncertainty markers. This means a model can output a perfectly formatted JSON file while silently altering critical clinical data.
- 1260 local inferences tested formatting stability.
- Mistral failed to parse up to 21.4% of notes without schema controls.
- Schema enforcement achieved 100% structural success across all models.
This finding forces health systems to separate structural validation from clinical validation. A clean JSON payload is not a safe clinical document. If an AI silently alters numeric values or clinical uncertainty while maintaining a perfect schema, it introduces hidden risks that automated parsers cannot detect. This evaluation relied entirely on automated proxy metrics, meaning no human clinicians reviewed the notes for medical safety.
Read the full analysis in Healthcare.