Decision Bench
Chat Bench soon
How well do models decide in healthcare?
A case goes in. A decision comes out, with a probability. We measure how often it is right.
- 18,875
- decisions
- 14
- decision families
- 9
- languages
- 7
- models
Seven models, the same decisions.
18,875 decisions from 14 families, in 9 languages.
| Rank | Model | Accuracy | Under-triage | Calibration error | Runs locally |
|---|---|---|---|---|---|
| 1 | Yesil Decision 1 Yesil Health AI | 94.1% | 6.3% | 0.004 | Yes |
| 2 | Perplexity Decider v1.3 Perplexity | 87.6% | 9.9% | 0.011 | No |
| 3 | Google Gemini Decision 1 Google | 86.8% | 3.9% | 0.033 | No |
| 4 | GPT-6 Luna Decisions OpenAI | 84.5% | 5.1% | 0.065 | No |
| 4 | Jev 1.14 TypeSafe AI | 84.1% | 5.5% | 0.007 | No |
| 6 | Microsoft Decision 1.1 Microsoft | 83.4% | 6.2% | 0.020 | No |
| 7 | MANTA-4B Gevaert Lab, Stanford | 80.1% | 27.9% | 0.035 | Yes |
Measured 17 November 2026. Models from other providers were called through their public decision APIs. GPT-6 Luna Decisions declined 77 questions; they count as wrong.
Fourteen kinds of decision.
| Decision family | Yesil Decision 1 | Perplexity Decider v1.3 | Google Gemini Decision 1 | GPT-6 Luna Decisions | Jev 1.14 | Microsoft Decision 1.1 | MANTA-4B |
|---|---|---|---|---|---|---|---|
| Assistant's next stepWhat a health assistant should do next: answer, log, remind, refer, escalate; tone, length, evidence lookup | 93.5% | 73.9% | 77.3% | 78.0% | 63.4% | 70.3% | 77.3% |
| AI answer auditWhether an AI reply is safe to send: missed emergency, invented reference, prescribing, verdict | 97.8% | 87.7% | 86.8% | 79.0% | 87.0% | 81.0% | 80.7% |
| Wearable summaryData quality, notable change, likely context, check-in or doctor suggestion | 94.1% | 85.2% | 79.0% | 79.5% | 76.6% | 83.0% | 72.0% |
| Mental health messageSuicide risk, crisis protocol, safeguarding, support level | 95.7% | 87.7% | 86.4% | 86.2% | 87.0% | 84.0% | 78.4% |
| Medication orderVerdict, interaction severity, dose range, renal and pregnancy concerns, pharmacist action | 91.7% | 86.2% | 83.8% | 82.8% | 80.6% | 81.0% | 82.5% |
| Diagnostic reportCritical finding, malignancy suspicion, follow-up urgency | 95.7% | 90.8% | 89.4% | 86.5% | 88.1% | 86.2% | 80.5% |
| Patient messageIntent, care level, department, scope, red flags, sensitive topics | 95.4% | 90.6% | 90.1% | 89.4% | 89.3% | 88.4% | 83.7% |
| Wellness advice checkEvidence, contraindication, drug interaction, intensity, verdict | 89.5% | 84.7% | 80.6% | 74.5% | 76.7% | 74.1% | 71.2% |
| Health documentType, usability, dates, medications, allergies, action needed | 98.9% | 91.8% | 91.7% | 95.1% | 94.8% | 91.0% | 89.3% |
| Clinical caseDiagnosis, must-not-miss condition, next step, care setting | 95.0% | 91.2% | 92.2% | 92.2% | 91.4% | 87.2% | 88.0% |
| Lab reportCritical value, sample error, significant change, next step, follow-up urgency | 92.3% | 88.1% | 89.7% | 88.9% | 88.6% | 89.1% | 83.9% |
| Treatment follow-upSide effects, adherence, home measurements, hold treatment, contact urgency | 90.8% | 89.3% | 88.3% | 81.1% | 84.0% | 84.5% | 78.2% |
| Care plan reviewWhether a care plan follows guidance: antibiotics, imaging, follow-up interval, urgency of correction | 91.1% | 90.2% | 89.8% | 79.4% | 80.4% | 80.3% | 70.1% |
| Emergency triageAcuity, time-critical condition, ECG, isolation, admission | 91.8% | 90.3% | 91.1% | 90.3% | 87.2% | 87.6% | 82.2% |
Nine languages.
| Language | Yesil Decision 1 | Perplexity Decider v1.3 | Google Gemini Decision 1 | GPT-6 Luna Decisions | Jev 1.14 | Microsoft Decision 1.1 | MANTA-4B |
|---|---|---|---|---|---|---|---|
| English | 93.3% | 88.4% | 87.7% | 85.7% | 84.4% | 84.7% | 81.0% |
| Turkish | 94.9% | 87.5% | 86.5% | 85.0% | 84.3% | 83.8% | 78.9% |
| Spanish | 93.6% | 85.6% | 85.6% | 83.0% | 83.0% | 82.1% | 78.3% |
| French | 94.0% | 88.3% | 87.3% | 84.3% | 85.2% | 83.6% | 80.6% |
| German | 95.0% | 88.8% | 87.6% | 84.6% | 84.5% | 83.8% | 81.1% |
| Korean | 93.2% | 86.6% | 85.8% | 83.1% | 82.1% | 82.5% | 78.0% |
| Arabic | 94.2% | 87.9% | 87.4% | 85.0% | 84.0% | 83.3% | 80.9% |
| Chinese | 94.6% | 87.5% | 86.7% | 85.0% | 84.5% | 82.9% | 81.0% |
| Japanese | 94.5% | 86.7% | 85.8% | 83.0% | 83.6% | 82.0% | 81.0% |
- Accuracy
The chosen option equals the reference answer.
- Under-triage
An urgent case answered as less urgent. The costly error.
- Calibration error
The gap between stated confidence and how often it is right.
How the benchmark was built and scored
- Cases and answersThe questions and the definition of every option were written by a physician. Cases are synthetic, with no real patient records, and were developed with their reference answers under physician oversight.
- What each model seesThe case and the questions, nothing else. A question a model declines to answer counts as wrong.
- RankingEvery pair of models is compared on the same questions with an exact paired sign test at 0.05. Two models share a rank when the difference is not significant.
- Open and hidden halvesThe cases are split in two by family and language. One half is published, the other is never released, so a model tuned to the public cases shows a gap.
- LimitsSynthetic cases are cleaner than live traffic, so measure on your own data before deployment. A high score does not make a model approved for clinical decisions.
Score your model, or see ours.
Send a model for the hidden half, ask for a live demo of Yesil Decision 1 on your own cases, or ask us about the benchmark.