← AI Health Hub

Research 7 min read

Decision Bench: how well do AI models decide in healthcare?

We scored seven AI decision models on 18,875 healthcare decisions in 14 decision families and 9 languages. Here is what we measured, how we built it, and what we found.

Yusuf Yesil, MD, MIS

Most of what AI does inside healthcare software is not writing answers. It is deciding.

A patient writes “my daughter has had a nosebleed since this morning” and the system has to decide how urgent that is. An assistant drafts a reply and something has to decide whether it is safe to send. A lab result arrives and something has to decide whether it is critical, or a bad sample. A medication order comes in and something has to decide whether a pharmacist should call. An agent finishes a turn and has to decide what to do next: answer, ask, look up evidence, or hand the case to a person.

These decisions happen thousands of times a day, quietly, before any human reads anything. Yet almost every public medical AI benchmark measures something else: exam knowledge, or the quality of a free-text answer. Neither tells you whether a model sends the right message to the emergency queue.

So we built a benchmark for the decisions themselves.

What Decision Bench measures

18,875 decisions across 2,479 cases, in 14 decision families and 9 languages (English, Turkish, Spanish, French, German, Arabic, Japanese, Korean and Chinese).

The 14 families cover the decisions that clinical and consumer health software makes every day:

FamilyWhat the model decides
Assistant’s next stepWhat a health assistant should do next: answer, log, remind, refer, escalate; tone, length, evidence lookup
AI answer auditWhether an AI reply is safe to send: missed emergency, invented reference, prescribing, verdict
Wearable summaryData quality, notable change, likely context, check-in or doctor suggestion
Mental health messageSuicide risk, crisis protocol, safeguarding, support level
Medication orderVerdict, interaction severity, dose range, renal and pregnancy concerns, pharmacist action
Diagnostic reportCritical finding, malignancy suspicion, follow-up urgency
Patient messageIntent, care level, department, scope, red flags, sensitive topics
Wellness advice checkEvidence, contraindication, drug interaction, intensity, verdict
Health documentType, usability, dates, medications, allergies, action needed
Clinical caseDiagnosis, must-not-miss condition, next step, care setting
Lab reportCritical value, sample error, significant change, next step, follow-up urgency
Treatment follow-upSide effects, adherence, home measurements, hold treatment, contact urgency
Care plan reviewWhether a care plan follows guidance: antibiotics, imaging, follow-up interval, urgency of correction
Emergency triageAcuity, time-critical condition, ECG, isolation, admission

Every question has a fixed set of options, a written definition for each option, and one reference answer. A model is scored on the decision it makes, not on how well it writes.

How it was built

The question catalogue (every decision, its wording, its options and the definition of each option) was written by a physician. The cases are synthetic: written for the benchmark, with no real patient records. Each case was written toward a target answer and then labelled blind by an independent pass. A question stays in the benchmark only when the two agree; where they disagreed, that question was removed from the case. Cases and reference answers were developed under physician oversight.

The benchmark is split in two halves, stratified by family and language:

  • The open half (1,239 cases, 9,422 questions) is on Hugging Face under CC BY 4.0, with a scorer that runs on the Python standard library.
  • The hidden half (1,240 cases, 9,453 questions) is never published. We run models on it on request and publish both halves side by side. A model tuned to the open half shows a gap between the two.

Three numbers, because accuracy is not enough

Accuracy. The top option equals the reference answer. Unanswered or refused questions count as wrong.

Under-triage. On ordered questions (care level, urgency scales), the share of answers placed below the reference. In healthcare this is the expensive error: an urgent case treated as routine. Two models with the same accuracy can differ a lot here.

Calibration. When a model says it is 90% sure, is it right 90% of the time? We report expected calibration error over ten confidence bins. Software that routes, escalates or asks a human depends on that number being honest.

Ranks come from an exact paired sign test: two models share a rank when the difference between them is not significant.

Results

We scored seven decision models on all 18,875 questions. Models from other providers were called through their public decision APIs on 17 November 2026.

RankModelAccuracyUnder-triageCalibration errorRuns locally
1Yesil Decision 1 (Yesil Health AI)94.1%6.3%0.004Yes, one 16 GB GPU
2Perplexity Decider v1.387.6%9.9%0.011No
3Google Gemini Decision 186.8%3.9%0.033No
4GPT-6 Luna Decisions (OpenAI)84.5%5.1%0.065No
4Jev 1.14 (TypeSafe AI)84.1%5.5%0.007No
6Microsoft Decision 1.183.4%6.2%0.020No
7MANTA-4B (Gevaert Lab, Stanford)80.1%27.9%0.035Yes, open weights

What stands out:

Yesil Decision 1 leads in all 14 families and all 9 languages. The gaps are largest where decisions depend on context rather than on a single fact: the assistant’s next step (+15.5 points over the next best model), AI answer audits (+10.1), wearable summaries (+8.9) and mental health messages (+8.0). The gaps are smallest in emergency triage (+0.8) and care plan review (+0.9), where every model is close to the ceiling set by how often careful readers agree.

It is also the best calibrated model (0.004). When it says 90%, it is right about 90% of the time.

It holds on the hidden half. 94.4% on the open half, 93.7% on the hidden half. Cases nobody has seen tell the same story.

It is not the lowest on under-triage. Google’s Gemini Decision places fewer urgent cases too low (3.9% against our 6.3%). That is the target for the next version of our model, and the benchmark is how we will measure it.

It runs on premises. Yesil Decision 1 runs on a single 16 GB GPU, so a hospital can keep patient data inside its own network. It answers in about 0.3 seconds per decision on an L4.

Why decisions deserve their own layer

A general model asked to “be careful” is slow, expensive and hard to audit. A decision model returns a typed answer with a probability, in a fraction of a second, every time in the same shape. That makes it a component you can build on:

  • A health chat app decides, before it writes a word, whether a question needs evidence from the literature, deeper reasoning, or an immediate safety message.
  • A patient inbox sorts thousands of messages by urgency and department before staff open them.
  • An AI assistant’s reply is checked for missed emergencies and invented sources before it reaches the patient.
  • A care team gets a nudge only when a wearable trend actually breaks from that person’s own baseline.

The probability matters as much as the decision. It is what lets software act alone when it is sure and ask a person when it is not.

Run it yourself

python score.py my_predictions.jsonl --json report.json

Limitations

  • Reference answers follow the option definitions written in each question. A model that applies a different but defensible clinical threshold will lose points on borderline cases.
  • Synthetic cases are cleaner than real messages and records. Treat these scores as an upper bound for the same model on live traffic, and measure on your own data before deployment.
  • The benchmark measures decisions, not medical knowledge in general and not the quality of free-text answers.
  • We built the benchmark and one of the models on it. That is why the scorer is open, half of the data is public, and any model can be run on the hidden half.
  • Not for clinical use. A high score does not make a model safe or approved for clinical decisions.

What comes next

Lower under-triage in the next model version. Chat Bench, a broader benchmark for health chat models, is in preparation. And a version of the decision model small enough to run on a phone.

If you build healthcare software and want to know how your model decides, or want Yesil Decision 1 inside your own network, write to us at contact@yesilscience.com.

This article is for informational purposes only and is not a substitute for professional medical advice, diagnosis or treatment.

Yusuf Yesil, MD, MIS
Yusuf Yesil, MD, MIS

Founder & Medical Doctor at Yesil Science. Clinical Biochemistry Specialist and Health AI researcher focused on clinical decision support, diagnostic AI, and biomedical machine learning.

Follow AI Health Hub