AI Decision Models Explained With Jev, Liquid d1 and New Benchmarks

A platform engineer runs every support ticket through a chat model with one prompt: billing, technical or account? Her code parses the JSON it writes back and pays for every output token. At her volume, the bill and the latency grow every month.

Since mid-September, a new class of model has been arriving for exactly this job. AI decision models return one option from a list plus a probability for each option, with no text to parse. One of them topped a public leaderboard at a fraction of the price, and her lead wants to know whether to swap it in by Friday.

The leaderboard number is real, but nobody measured it on her tickets. This post covers what decision models are and the launches that started the category. It then shows how to read the Artificial Analysis Decision Index and how to test a candidate on your own labelled traces.

TL;DR

• AI decision models return a typed answer (yes or no, one option from a list, or a point on a scale) with a probability for each option, instead of text.
• TypeSafe AI opened early access to Jev on September 15, 2026; Liquid AI introduced d1 on October 5 and Microsoft shipped Microsoft-Decision-1 on October 9.
• Jev 1.13 led the Decision Index v1.0 (October 13); v2 (November 8) rebuilt the index around calibrated Decision Scores, and Drex Max v1.1 now leads its decision models.
• They fit high-volume gates, classification, routing and first-pass judging.
• A public score is measured on someone else’s questions, so test accuracy, calibration and answer flips on your own labelled traces first.

What AI decision models are and how they differ from chat models

AI decision models answer a bounded question in one call. You send the input and a typed question, and the model returns one answer from the options you defined plus a probability for each option. A chat model writes free text that your code must parse, and it bills you for every token it writes.

TypeSafe and Liquid AI name the three question shapes Noul (yes or no), Choice (one option from a list) and Score (a point on a scale), and Microsoft’s API uses noul, choice and score. Nothing is generated, so TypeSafe and Liquid bill input only. Jev costs $0.042 per million input tokens with free output, as of November 11, 2026.

A chat model in JSON mode is a different thing. It still writes a string token by token that you must parse and validate. Decision Index v2 even asks language models for “one JSON object that gives a probability for every option”, where v1.0 let them name one option. A decision model returns that distribution natively.

The category starts with TypeSafe’s launch post for System One models, which frames Jev as unstructured state in and typed probabilistic decisions out.

Typed answers with a probability attached

The per-option probability is what makes these models useful. Software acts above a threshold and escalates below it. TypeSafe’s homepage says “Set the thresholds for when it acts autonomously and when it asks for review”, and Microsoft’s deployment guide adds “Use a low top probability as a signal to escalate the decision to a person or another model.”

That turns one model call into a triage step. Confident answers flow straight through, and uncertain ones go to a stronger model or a human. A chat model that names a single option with no probabilities gives your code nothing to draw that line with, so every answer gets the same trust.

Why TypeSafe calls Jev a System One model

TypeSafe avoids the decision model label for its own product. Its docs say “Jev is TypeSafe’s flagship model and the first System One model”, a name drawn from Daniel Kahneman’s fast System 1 and slow System 2 thinking. Liquid AI, Microsoft and Artificial Analysis all say decision model, so this post does too.

On November 3, 2026, TypeSafe’s case against the decision model label argued that “Jev makes judgments, code makes decisions”, meaning your code should hold the business rules and ask the model only the judgment calls.

Three launches that turned AI decision models into a category

Three releases in four weeks gave AI decision models a market. TypeSafe AI opened early access to Jev on September 15, 2026. Liquid AI introduced d1 on October 5 after an experimental release the week before. Microsoft released Microsoft-Decision-1 in Microsoft Foundry on October 9, with the same state and typed-question pattern.

None of the three leads the index today. On the live Decision Index v2 leaderboard as of November 11, 2026, Jev 1.14 scores 67.0, Microsoft-Decision-1 62.8 and d1 61.6, while Liquid’s newer d1-ultra reaches 70.4. Scores move with every release, so treat them as a dated snapshot.

Jev, the TypeSafe AI model that opened the category

Jev was trained with what TypeSafe calls Reinforcement Learning for Calibrated Decisions (RLCD) and ships no public weights. The docs list jev-1.13.0 and jev-1.14.0 (released October 14), both with a 64k per-request and 32k per-question context limit. The “193.6x Faster, 444.6x Cheaper” headline on TypeSafe’s homepage comes from its own workflow evals.

Liquid AI decision models, from d1 to d1-ultra

Liquid AI’s d1 reads text and images through the Liquid API at the path decisions/v1/systemone. Open d1-3B and d1-omni-600M weights followed on October 7 and hosted d1-ultra on October 15. Liquid’s docs now recommend d1-2B (1.9B parameters) and list d1-omni-2B (2.2B, adds audio), both on Hugging Face since October 24.

Liquid AI’s d1 announcement lists d1 on OpenRouter and Vercel AI Gateway, and the docs now add d1-ultra to both.

Read Liquid’s d1-ultra claim with its date attached. On October 15, Liquid said d1-ultra beat every other decision model on the Artificial Analysis index by at least 8 points, while that index was still version 1.0. On the rebuilt v2 leaderboard, d1-ultra scores 70.4, behind Drex Max v1.1 at 73.4.

Microsoft-Decision-1 in Microsoft Foundry

Microsoft’s Decision-1 launch post says “we post trained Qwen3.5-9B for fast, single-pass decision scoring” and that “a 90% prediction should be right about nine times out of 10 on representative cases.”

Microsoft’s Decision-1 deployment guide is unusually candid about the model’s limits.

It targets “classification, routing, ranking, grading, and binary decisions” and returns no written rationale. It tells you to “validate it against labeled examples before you automate a gate”, and it warns that “Calibration is strongest on familiar task types.” Both lines matter later in this post, when the leaderboard meets your own data.

What the Artificial Analysis Decision Index measures

The Artificial Analysis Decision Index is the independent decision model benchmark that scores all of these releases on the same tests. Version 1.0, published October 13, 2026, averaged knowledge, reasoning, tool-calling and safety-classification tests. Version 2, published November 8, adds a private document benchmark and scores calibrated confidence alongside raw accuracy.

Version 1.0 at launch on October 13

Jev 1.13 led version 1.0 at 58, ahead of GLiDE and Microsoft-Decision-1. Claude Haiku 5.5 (Low) scored 61, but was 11 times slower than Jev 1.13 and cost 4.4 times as much. Liquid’s 48.57 for d1-3B comes from a different index, the community Decision Index v0.2.1 (public split) on Hugging Face.

The full component breakdown is in the Decision Index v1.0 launch results.

Version 2 and the Decision Score

Version 2 added AA-Paperclip (1,500 private documents, each scored as one decision over 15 answers) and removed WildGuardTest “given its high saturation.” Each question now scores 50 x (1 + c - (p - c)^2), where p is the chosen answer’s probability and c is 1 if right and 0 if wrong. Confident mistakes cost the most.

The Decision Index leaderboard page calls that score “the payoff of answering when p is above a threshold and routing the question elsewhere otherwise, averaged over every threshold”, and what changed in Decision Index v2 covers each new benchmark.

As of November 11, 2026, Drex Max v1.1 leads the decision models at 73.4, ahead of GLiDE v1.2 (72.8), Mammoth 27B (72.4), Decider v1.2 (70.5), d1-ultra (70.4), SubQ Snap 1.1 (69.2) and Elephant 9B (67.5). Jev 1.14 ranks eighth of 17. Microsoft-Decision-1 is 12th and d1 14th.

Reading the decision model benchmark leaderboard

Read a decision model benchmark leaderboard as four separate numbers. The index score blends accuracy with calibration. Cost per million tasks and time per task tell you what a gate costs at volume. The calibration view tells you whether a 0.9 probability is right about nine times in ten.

Open the live Decision Index leaderboard and switch between its Score, Accuracy, Calibration, Time and Cost tabs.

Turn on the “Include language models” toggle and the Max settings of Claude Haiku 5.5 (77.1) and GPT-6 Luna (76.4) top every decision model. Luna Max costs $1,530 per million tasks against $102 for Drex Max v1.1. Jev 1.14 answers in 92 milliseconds, while Claude Haiku 5.5 at Max takes 32.65 seconds.

Score, cost and time per task

Cost per million tasks is not token price. Decision models bill input tokens only, but v2 found that GLiDE v1.2 bills a document’s input once per question, so it costs $780 per million tasks against $102 for Drex Max v1.1. A gate that runs on every request should sit near the cost chart’s Pareto line.

Calibration and the Brier score

The Brier score is the mean squared gap between a model’s probability for its chosen answer and whether that answer was right. It runs from 0 to 1, and lower is better. Version 1.0 found that “accuracy falls below that probability for all decision models” at almost every probability level, so 0.9 is not a 90% promise.

Calibration can move a ranking on its own. On AA-Paperclip, Celeris-1.1 Decision passes 24% of documents to Clef’s 36%, yet ties on Decision Score because its Brier score is 0.15 against 0.27. The calibration chart plots accuracy against stated probability in bins of 0.1, each holding at least 20 answers.

Where decision models fit in a production stack

Decision models fit where software needs a bounded answer many times a second. Ticket triage, content gates, intent classification, routing a request to a tool or model, and first-pass grading of agent outputs all qualify. Writing, summarising or explaining still belongs to a generative model, often in the same pipeline.

Gates, triage and routing decisions

One support message on the Artificial Analysis page shows the shape: a Choice for billing, technical or account, a yes or no on urgency and a Score for how upset the customer is. All three go in one call. Microsoft adds: “Use Microsoft-Decision-1 to route or gate a request, and then send approved requests to a generative model.”

First-pass judging of agent outputs

Grading agent outputs follows the same pattern. A decision model answers a typed question about a response, such as whether it followed a policy, and a low probability sends the case to a stronger judge or a person. Liquid’s docs list LLM-as-judge replacement among its scoring uses, and Microsoft names evaluating model output.

For the judge workflow itself, read using Jev as a judge for agent evals, which compares typed judges with LLM-as-a-judge and code checks.

Decision models do not fit open-ended work. A summary, a reply, an explanation or code still needs a generative model. So does any task where a user must see the reasoning, because Microsoft’s guide states plainly that its model returns numerical decisions without a written rationale.

Why a public leaderboard score will not transfer to your task

A Decision Index score averages someone else’s questions, option sets and wording. Your tickets, labels and edge cases are different, so the ranking can change on your data. Calibration also drifts, because a model’s probabilities are best on task types it saw in training and weaker on yours.

First, the rank depends on what gets measured. Within four weeks, a new headline metric, new benchmarks and new versions moved the top spot from Jev 1.13 to Drex Max v1.1. Second, calibration was weakest on knowledge and reasoning in version 1.0, and Microsoft says its model calibrates best on familiar task types.

Third, wording changes answers. AA-Symmetry rewords questions by reordering options, swapping labels or negating yes or no questions, and in v2 Drex Max v1.1 kept the same answer on 91% of them and Clef on 67%. Fourth, vendor charts are self-reported: TypeSafe notes its own team built its workflow evals, “so some bias could exist.”

Before you pick a model at all, check when a deterministic check beats a model judge, because some gates need only a rule.

Your labels, your options, your wording

The option list you write is part of the prompt. Rename “Billing” to “Payments”, add an “Other” bucket or reorder the choices, and you have built a test the index never ran. Use the public score to shortlist candidates, then run the test below on your own data.

Testing an AI decision model on your own labelled traces

Test an AI decision model the way you would test a new hire on real work. Pull a few hundred production traces and label the right answer for each. Run the candidate next to your current model and compare accuracy, calibration, answer flips, cost and time before any gate goes live.

1. Sample traces across your real traffic mix, including rare classes.
2. Label each with the exact option set you will ship.
3. Run the candidate and your current baseline on identical inputs.
4. Measure accuracy, a reliability table in 0.1 bins and a Brier score.
5. Re-ask a slice with reordered, relabelled or negated options and count flips.
6. Choose the threshold where auto-accept and error rates both meet your bar, and escalate the rest.
7. Re-run on every model version change, since jev-latest points to jev-1.14.0 as of November 11, 2026.

Build a labelled set from real traces

Oversample rare classes so each one has enough examples to measure, and write the labelling rules down so two people label the same ticket the same way. Use the exact option labels you will ship. Hold back a slice you never look at while tuning, and score the final candidate on it once.

Measure accuracy, calibration and flip rate

Read the three numbers together. A model with high accuracy and poor calibration forces a high threshold, so it escalates more traffic than its accuracy suggests. A model with a high flip rate breaks the day someone edits an option label. Compare each number against your current prompt on the same inputs.

Pick a confidence threshold you can defend

Sweep the threshold from 0.5 to 0.99 and record the share of traffic answered alone and its error rate. Pick the point where both meet your bar and write it down per gate. Everything below goes to a stronger model or a person, and a pinned model version keeps the numbers stable.

If you want this comparison inside an eval platform, Future AGI Evaluation offers Jev as an evaluator model with your own TypeSafe key, pinned to jev-1.13.0 or tracking jev-latest. You pick a Pass/Fail, Choices or Score output. Then attach it to a dataset of labelled traces, so its typed verdicts sit beside your labels.

Setup steps and output types are on the Jev evaluators in Future AGI Evaluation page.

For cases below your threshold, compare stronger judge models for the escalation path.

Choosing your first decision model gate

Start with one gate where a wrong answer is cheap to catch, such as ticket routing. Label a sample of last month’s tickets and run your current prompt and one decision model on them. Swap only if the candidate matches accuracy, stays calibrated at your threshold and costs less per task.

That gives the platform engineer a better answer for Friday than yes or no. She runs her prompt and the leaderboard pick on last week’s labelled tickets and finds the probability above which the decision model is right as often as her prompt. Above that line it answers alone, and below it the chat model takes over.

Keep the labelled set, the threshold and the test, because another release will land soon. Three vendors shipped in four weeks and the index rebuilt itself within a month. With the test in place, judging each new model takes an afternoon of measurement.