Decision Bench Chat Bench soon

How well do models decide in healthcare?

A case goes in. A decision comes out, with a probability. We measure how often it is right.

18,875
decisions
14
decision families
9
languages
7
models

Seven models, the same decisions.

18,875 decisions from 14 families, in 9 languages.

RankModelAccuracyUnder-triageCalibration errorRuns locally
1Yesil Decision 1 Yesil Health AI94.1%6.3%0.004Yes
2Perplexity Decider v1.3 Perplexity87.6%9.9%0.011No
3Google Gemini Decision 1 Google86.8%3.9%0.033No
4GPT-6 Luna Decisions OpenAI84.5%5.1%0.065No
4Jev 1.14 TypeSafe AI84.1%5.5%0.007No
6Microsoft Decision 1.1 Microsoft83.4%6.2%0.020No
7MANTA-4B Gevaert Lab, Stanford80.1%27.9%0.035Yes

Measured 17 November 2026. Models from other providers were called through their public decision APIs. GPT-6 Luna Decisions declined 77 questions; they count as wrong.

Accurate and safe are two different things.Higher is more accurate. Further left, fewer urgent cases are treated as routine.
The hidden half tells the same story.Half of the cases are public, half are kept back. Accuracy on each.

Fourteen kinds of decision.

Nine languages.

When it says 90%, it is right 90% of the time.Stated confidence against how often the answer was right. A model on the diagonal can be trusted at its word.
  1. Accuracy

    The chosen option equals the reference answer.

  2. Under-triage

    An urgent case answered as less urgent. The costly error.

  3. Calibration error

    The gap between stated confidence and how often it is right.

How the benchmark was built and scored
  • Cases and answersThe questions and the definition of every option were written by a physician. Cases are synthetic, with no real patient records, and were developed with their reference answers under physician oversight.
  • What each model seesThe case and the questions, nothing else. A question a model declines to answer counts as wrong.
  • RankingEvery pair of models is compared on the same questions with an exact paired sign test at 0.05. Two models share a rank when the difference is not significant.
  • Open and hidden halvesThe cases are split in two by family and language. One half is published, the other is never released, so a model tuned to the public cases shows a gap.
  • LimitsSynthetic cases are cleaner than live traffic, so measure on your own data before deployment. A high score does not make a model approved for clinical decisions.

Score your model, or see ours.

Send a model for the hidden half, ask for a live demo of Yesil Decision 1 on your own cases, or ask us about the benchmark.

What do you need?

We use your details only to reply to this request.