Most of what AI does inside healthcare software is not writing answers. It is deciding.
A patient writes “my daughter has had a nosebleed since this morning” and the system has to decide how urgent that is. An assistant drafts a reply and something has to decide whether it is safe to send. A lab result arrives and something has to decide whether it is critical, or a bad sample. A medication order comes in and something has to decide whether a pharmacist should call. An agent finishes a turn and has to decide what to do next: answer, ask, look up evidence, or hand the case to a person.
These decisions happen thousands of times a day, quietly, before any human reads anything. Yet almost every public medical AI benchmark measures something else: exam knowledge, or the quality of a free-text answer. Neither tells you whether a model sends the right message to the emergency queue.
So we built a benchmark for the decisions themselves.
What Decision Bench measures
18,875 decisions across 2,479 cases, in 14 decision families and 9 languages (English, Turkish, Spanish, French, German, Arabic, Japanese, Korean and Chinese).
The 14 families cover the decisions that clinical and consumer health software makes every day:
| Family | What the model decides |
|---|---|
| Assistant’s next step | What a health assistant should do next: answer, log, remind, refer, escalate; tone, length, evidence lookup |
| AI answer audit | Whether an AI reply is safe to send: missed emergency, invented reference, prescribing, verdict |
| Wearable summary | Data quality, notable change, likely context, check-in or doctor suggestion |
| Mental health message | Suicide risk, crisis protocol, safeguarding, support level |
| Medication order | Verdict, interaction severity, dose range, renal and pregnancy concerns, pharmacist action |
| Diagnostic report | Critical finding, malignancy suspicion, follow-up urgency |
| Patient message | Intent, care level, department, scope, red flags, sensitive topics |
| Wellness advice check | Evidence, contraindication, drug interaction, intensity, verdict |
| Health document | Type, usability, dates, medications, allergies, action needed |
| Clinical case | Diagnosis, must-not-miss condition, next step, care setting |
| Lab report | Critical value, sample error, significant change, next step, follow-up urgency |
| Treatment follow-up | Side effects, adherence, home measurements, hold treatment, contact urgency |
| Care plan review | Whether a care plan follows guidance: antibiotics, imaging, follow-up interval, urgency of correction |
| Emergency triage | Acuity, time-critical condition, ECG, isolation, admission |
Every question has a fixed set of options, a written definition for each option, and one reference answer. A model is scored on the decision it makes, not on how well it writes.
How it was built
The question catalogue (every decision, its wording, its options and the definition of each option) was written by a physician. The cases are synthetic: written for the benchmark, with no real patient records. Each case was written toward a target answer and then labelled blind by an independent pass. A question stays in the benchmark only when the two agree; where they disagreed, that question was removed from the case. Cases and reference answers were developed under physician oversight.
The benchmark is split in two halves, stratified by family and language:
- The open half (1,239 cases, 9,422 questions) is on Hugging Face under CC BY 4.0, with a scorer that runs on the Python standard library.
- The hidden half (1,240 cases, 9,453 questions) is never published. We run models on it on request and publish both halves side by side. A model tuned to the open half shows a gap between the two.
Three numbers, because accuracy is not enough
Accuracy. The top option equals the reference answer. Unanswered or refused questions count as wrong.
Under-triage. On ordered questions (care level, urgency scales), the share of answers placed below the reference. In healthcare this is the expensive error: an urgent case treated as routine. Two models with the same accuracy can differ a lot here.
Calibration. When a model says it is 90% sure, is it right 90% of the time? We report expected calibration error over ten confidence bins. Software that routes, escalates or asks a human depends on that number being honest.
Ranks come from an exact paired sign test: two models share a rank when the difference between them is not significant.
Results
We scored seven decision models on all 18,875 questions. Models from other providers were called through their public decision APIs on 17 November 2026.
| Rank | Model | Accuracy | Under-triage | Calibration error | Runs locally |
|---|---|---|---|---|---|
| 1 | Yesil Decision 1 (Yesil Health AI) | 94.1% | 6.3% | 0.004 | Yes, one 16 GB GPU |
| 2 | Perplexity Decider v1.3 | 87.6% | 9.9% | 0.011 | No |
| 3 | Google Gemini Decision 1 | 86.8% | 3.9% | 0.033 | No |
| 4 | GPT-6 Luna Decisions (OpenAI) | 84.5% | 5.1% | 0.065 | No |
| 4 | Jev 1.14 (TypeSafe AI) | 84.1% | 5.5% | 0.007 | No |
| 6 | Microsoft Decision 1.1 | 83.4% | 6.2% | 0.020 | No |
| 7 | MANTA-4B (Gevaert Lab, Stanford) | 80.1% | 27.9% | 0.035 | Yes, open weights |
What stands out:
Yesil Decision 1 leads in all 14 families and all 9 languages. The gaps are largest where decisions depend on context rather than on a single fact: the assistant’s next step (+15.5 points over the next best model), AI answer audits (+10.1), wearable summaries (+8.9) and mental health messages (+8.0). The gaps are smallest in emergency triage (+0.8) and care plan review (+0.9), where every model is close to the ceiling set by how often careful readers agree.
It is also the best calibrated model (0.004). When it says 90%, it is right about 90% of the time.
It holds on the hidden half. 94.4% on the open half, 93.7% on the hidden half. Cases nobody has seen tell the same story.
It is not the lowest on under-triage. Google’s Gemini Decision places fewer urgent cases too low (3.9% against our 6.3%). That is the target for the next version of our model, and the benchmark is how we will measure it.
It runs on premises. Yesil Decision 1 runs on a single 16 GB GPU, so a hospital can keep patient data inside its own network. It answers in about 0.3 seconds per decision on an L4.
Why decisions deserve their own layer
A general model asked to “be careful” is slow, expensive and hard to audit. A decision model returns a typed answer with a probability, in a fraction of a second, every time in the same shape. That makes it a component you can build on:
- A health chat app decides, before it writes a word, whether a question needs evidence from the literature, deeper reasoning, or an immediate safety message.
- A patient inbox sorts thousands of messages by urgency and department before staff open them.
- An AI assistant’s reply is checked for missed emergencies and invented sources before it reaches the patient.
- A care team gets a nudge only when a wearable trend actually breaks from that person’s own baseline.
The probability matters as much as the decision. It is what lets software act alone when it is sure and ask a person when it is not.
Run it yourself
- Page and full results (by family and by language): yesilscience.com/bench/decision
- Open half, scorer and our predictions: huggingface.co/datasets/yesilhealth/Yesil_Decision_Bench
- Have a model scored on the hidden half: use the form at the end of the bench page. We run it and publish both halves.
python score.py my_predictions.jsonl --json report.jsonLimitations
- Reference answers follow the option definitions written in each question. A model that applies a different but defensible clinical threshold will lose points on borderline cases.
- Synthetic cases are cleaner than real messages and records. Treat these scores as an upper bound for the same model on live traffic, and measure on your own data before deployment.
- The benchmark measures decisions, not medical knowledge in general and not the quality of free-text answers.
- We built the benchmark and one of the models on it. That is why the scorer is open, half of the data is public, and any model can be run on the hidden half.
- Not for clinical use. A high score does not make a model safe or approved for clinical decisions.
What comes next
Lower under-triage in the next model version. Chat Bench, a broader benchmark for health chat models, is in preparation. And a version of the decision model small enough to run on a phone.
If you build healthcare software and want to know how your model decides, or want Yesil Decision 1 inside your own network, write to us at contact@yesilscience.com.




