Off-the-shelf artificial intelligence tools fail to extract complex psychiatric symptoms from Spanish clinical notes without extensive retraining.
Mining free-text health records is critical for large-scale psychiatric research. Yet off-the-shelf language models struggle with non-English clinical jargon and local idioms. Relying on untreated open-source models risks missing critical symptom context altogether.
That baseline failure exposes a wider problem in digital health. Without localized tuning, massive generative engines perform no better than coin flips when pulling granular data from regional health records.
fine-tuning closes the performance gap
Researchers evaluated 11 open-source generative large language models against a traditional natural language processing method across 109 psychiatric concepts. The data came from annotated electronic health records at a psychiatric hospital in Colombia.
Out-of-the-box performance was weak. Zero-shot approaches achieved a macro-F1 score of just 0.2 to 0.6, while few-shot prompting reached 0.2 to 0.74. Fine-tuning dramatically elevated accuracy to a macro-F1 of 0.75 to 0.86, allowing several open models to outperform traditional rule-based pipelines. In total, 100 psychiatric concepts were reliably detected with an F1 score above 0.8.
- Base open-source models scored poorly without direct training (0.2-0.6 zero-shot macro-F1).
- Fine-tuning boosted accuracy significantly, topping out at 0.86 macro-F1.
- Synthetic training data yielded a shareable model scoring a 0.79 macro-F1.
- Cross-hospital transfer caused only slight performance loss (0.02-0.16 macro-F1 drop).
Similar challenges exist across other non-English domains. A study on oncology data extraction from Spanish records showed that tailored language tools are essential for real-world electronic health records. Other research on multilingual biomedical concept normalization confirms that raw models need structured local adaptation to work safely.
Privacy-preserving synthetic data succeeds
Sharing clinical models across health systems usually stalls over privacy laws and patient risk. To solve this, the team generated synthetic clinical text paired with real concept annotations to build an open-source model named Mistral-small-psych. This privacy-safe engine matched real-world training, reaching a 0.79 macro-F1 score.
When tested at a second independent hospital in Colombia, the fine-tuned models held up well. Out-of-domain evaluation showed minimal degradation, with macro-F1 scores dropping by only 0.02 to 0.16 across sites.
This validates a clear operational path forward. Hospitals do not need to expose raw patient records to external developers to build accurate clinical software.
Real limitations remain
These findings show technical feasibility, not immediate clinical automation. Extracted concept flags aid cohort building for research, but retrospective validation at two Colombian sites does not mean the software is ready to act directly on patient care without physician review.
Health systems looking to mine Spanish records should avoid off-the-shelf deployments. They must invest in localized synthetic fine-tuning before trusting automated symptom pipelines.
Read the full paper in the Journal of Medical Internet Research.
