A new clinical benchmark reveals that large language models fail frontline health workers unless their queries are translated into English first.

Can an AI save a life if it cannot understand the patient’s voice? Silicon Valley builds clinical models on clean English text. Yet real-world medicine is messy, multilingual, and spoken. This disconnect is where global health tech initiatives usually fall apart.

A new preprint evaluating LLMs in Nigeria exposes a massive gap between laboratory promise and point-of-care reality. The study challenges the assumption that we can deploy off-the-shelf models globally without heavy localization. It suggests that the front-end translation pipeline is actually more critical than the underlying clinical model itself.

The translation bottleneck

Researchers introduced NigBench to test these systems in real-world environments. The dataset contains over 9,000 real-world, point-of-care, multilingual clinical question-answer pairs. These questions came directly from frontline health workers in Nigeria. The study compared local general practitioners to multiple leading open and closed LLMs across text and speech modalities. While human doctors easily navigate local dialects, the machines faltered.

The results show that language barriers degrade AI performance faster than clinical complexity does. This finding aligns with broader research on the architecture needed for inclusive healthcare guidance, which emphasizes that raw model size cannot overcome poor local language processing.

How the models performed

• Models achieved their highest accuracy when processing English text inputs.
• Performance dropped significantly when models were tested with local-language speech.
• Pre-transcribing and translating local speech into English before prompting created substantial performance gains.
• Small language models (SLMs) exhibited key limitations that make them unsuitable for frontline clinical support in these settings.

This is a wake-up call for developers. The active ingredient in clinical AI deployment is not the reasoning engine. It is the translation layer.

This gap is the real story. It proves that clinical knowledge is useless without linguistic accessibility. If an app cannot accurately capture spoken Hausa, Igbo, or Yoruba, the clinical accuracy of the LLM is irrelevant. This complicates the push for lightweight, offline models. It suggests that developers must prioritize robust speech-to-text pipelines over larger medical datasets.

The path to trust

We must be honest about the limits of this research. The study is a preprint and focuses on a single country. These translation workarounds may not scale easily to regions with different dialect densities.

However, the lesson is clear. To build trustworthy clinical artificial intelligence, we must stop treating translation as an afterthought. Until speech processing catches up, English remains the gatekeeper of medical AI.

Read the full study on medRxiv.