A new study shows that smart prompting makes cheap, small AI models triage patients just as safely as expensive ones.

Why do we pay senior specialists to sort through paperwork? In rheumatology, triage is a high-volume administrative bottleneck that wastes expert hours on sorting rather than treating. Human triage systems are only moderately accurate and notoriously hard to reproduce.

This study challenges the assumption that clinical AI requires massive, expensive proprietary models. By using advanced prompting, cheap models can perform at a specialist level. This shifts the economic equation of clinical AI deployment.

Testing the Models

Researchers built 20 real-world referral scenarios spanning the urgency spectrum. Four Australian rheumatologists established a consensus baseline. The study evaluated 23 large language models, tasking them with sorting each referral into one of 5 urgency tiers. Each model ran the task three times, generating 2,760 total decisions across simple and advanced prompting conditions.

Under the simple prompt, performance scaled with size and cost. Larger models dominated, with a Spearman correlation of 0.42 (P = .047) between model size and accuracy. This baseline confirms that raw computational power usually dictates clinical accuracy when instructions are basic.

Advanced prompting changed everything. By providing explicit expectations and worked examples, researchers shrank the variance in accuracy between different models 5.3-fold, dropping from 0.014 to 0.003 (P = .01). Suddenly, the link between model size and accuracy vanished (rho = -0.05; P = .83), and cost no longer predicted performance.

The Triage Results

• All 2,760 attempts successfully returned valid urgency categories.
• Leading models matched or exceeded the accuracy range of human specialists.
• Advanced prompting substituted for the raw reasoning capability of larger, expensive models.

The Real-World Implications

This finding aligns with emerging research on clinical workflows. For instance, research on rheumatic disease diagnosis highlights how task-shifting can optimize hybrid care. Similarly, using AI to generate referral summaries for co-management shows that the administrative burden in rheumatology is ripe for automation.

The clinical takeaway is economic. Health systems do not need to license the most expensive commercial APIs to automate administrative sorting. A smaller, self-hosted model with robust prompting can achieve equivalent safety and accuracy. However, the persistence of under-triage errors means these systems cannot run on autopilot yet. They must serve as a first-pass filter, not the final judge.

Read the full study on medRxiv.