Open-weight AI models can extract clinical comorbidities from medical text just as accurately as proprietary systems like GPT-4 and Claude.
Hospitals often hesitate to send patient notes to commercial cloud APIs due to privacy rules. Running open-weight models on local servers solves the privacy problem, but past open models lacked precision. This new benchmark upends that trade-off, demonstrating that locally hosted tools no longer require a drop in performance for phenotype extraction.
That shift matters for institutional adoption. It confirms that health systems can bypass external API fees and third-party data processing agreements without losing analytical rigor, accelerating how automated data pipelines support clinical trial matching and risk stratification.
Benchmarking 46 Models Across Notes
A multi-institution study evaluated 46 language models on the task of labeling 380 English clinical notes from two US institutions. A radiology resident and a medical student, supervised by a senior radiologist, manually annotated the notes to establish a reference standard for 29 conditions using the simplified Elixhauser Comorbidity Index. Researchers then used the GPT-o3-mini-high reasoning model to convert free-text model outputs into binary decisions to compute recall, precision, and F1-scores.
The evaluation spanned 4 commercial models from the GPT and Claude lines alongside 42 open-weights models, including 13 Llama models and 29 Qwen models. Top open-weights performers—such as Llama 3 70B, Llama 3.1 70B, Llama 3.1 405B, Llama 3.3 70B, Qwen 1.5 72B, Qwen 1.5 110B, Qwen 2 72B, and Qwen 2 VL 72B—all crossed an F1-score of 70%, matching proprietary baselines. Statistical modeling revealed that recency and parameter size primarily drove gains in precision rather than recall.
This finding aligns with broader trends in generative artificial intelligence for automated data extraction, which highlight the capability of modern architectures to parse unstructured EHR text. It also builds upon classic research in electronic health records-driven phenotyping by providing a viable, private alternative to manual chart review.
Key Trial Outcomes
- Top open-weight models like Llama 3.3 70B achieved F1-scores above 70%, performing on par with commercial systems.
- Evaluating 29 individual comorbidities revealed significant performance variance depending on the specific condition.
- Model capacity and release recency yielded statistically significant improvements in precision across the 46 models tested.
Limits and Next Steps
The study relies on a retrospective sample of 380 notes, which limits generalizability across diverse clinical specialties. Additionally, extracting comorbidity labels from text does not measure downstream clinical outcomes or workflow integration directly.
Health systems should run prospective pilots using local 70B-parameter open-weights models for low-risk administrative workflows before replacing manual abstraction entirely.
Read the full study in the Journal of Medical Internet Research.
