Best LLM Judge Models in 2026: 8 Models Ranked on Calibration, Cost, and Self-Preference

You ship a customer-support agent. The LLM judge is GPT-4o and the helpfulness rubric reads 0.91 every Monday. In March the judge bumps to a 4o minor version. In April the agent quotes a refund off by an order of magnitude, and the rubric still reads 0.91.

The signal stopped meaning what you thought it meant the day the judge changed. A single SummEval Spearman or MT-Bench winrate is a thin basis for ranking judge models. Those numbers do a lot of unspoken work. A judge that hits 0.514 Spearman on summarization is not automatically the judge that holds for two years on your customer-support rubric.

The benchmark measures one task on one dataset with one rubric the paper authors wrote. The thesis this post defends: pick a judge model on three axes, which are human correlation against your rubric, cost per score at your traffic volume, and self-preference bias against your candidate models. The best judge for your eval wins on your rubric, not on SummEval.

We compare eight judge models worth running on a calibration set in October 2026, plus the open-source judge models built only for grading. Each one gets the same treatment: where it earns its bill, what a score costs, and how much it favors its own family.

Methodology note: scoring axes below are calibration ceiling (kappa against human labels on subjective rubrics), cost per million tokens, self-preference bias documented across published work, and license/deployment shape. Pricing verified October 2026 against vendor pricing pages; some prices are promotional and change on Dec 1, 2026 (Gemini 4 Argon) and Jan 1, 2027 (Gemini 3.8 Flash). Calibration is task-dependent. Treat these as starting points, not procurement decisions.

Which LLM judge model wins on which axis?

If you read one row: there is no single winner. Run two or three candidates on a 100-to-300 example human-labeled set, measure kappa, divide by cost per score, then make the call.

How do you pick the best model for LLM-as-a-judge?

A published leaderboard collapses the choice to one number. In production, three axes bind separately and the binding axis changes by team.

Human correlation on your rubric. Cohen’s kappa or Spearman against a human-labeled hold-out, computed on your data, not on SummEval. A low-stakes marketing-copy rubric can live with lower agreement than a medical-advice rubric, so set the kappa bar per rubric before you test.

A judge that hits 0.85 on a public benchmark and 0.55 on your dataset is a judge that does not know your domain. The fix is unglamorous: hand-label 100 to 300 examples covering the failure modes that matter, run every candidate judge against the same set, and only then read the leaderboard.

Cost per score at your traffic volume. If each frontier judge call costs $0.01 to $0.05, a million traces a day is $300K to $1.5M a month. The judge that wins on kappa and loses on dollar-per-score loses the procurement.

Three patterns rescue the bill. Small judges such as Claude Haiku 5.5 or GPT-6 Luna cost about $0.0003 per 2K-token score against about $0.006 on Sonnet 5.5 or GPT-6.1 Sol. Classifier cascades escalate only close cases to the frontier model, and sample-don’t-score covers routine traffic.

Self-preference bias against your candidates. In the original MT-Bench study (Zheng et al. 2023), GPT-4 gave its own answers a 10% higher win rate and Claude-v1 a 25% higher one, though the authors called the evidence inconclusive. Measure it, do not assume it.

The cardinal mistake is the same model as judge and candidate. The second mistake is judging GPT candidates with a GPT judge and Claude candidates with a Claude judge in the same eval suite, which builds family bias into the comparison. The mitigation is a three-judge ensemble across families on launch decisions and single-family judges only for trend tracking.

Add one operational axis underneath the three: judge version stability. A minor judge version bump (for example Sonnet 4.5 to 4.6) can shift rubric scores, so pin the exact judge version and recalibrate on your labeled set after any upgrade. See why LLM-as-a-judge and G-Eval definitive guide for the contract pattern.

The best LLM judge models in 2026, ranked

1. Future AGI Turing models (turing_large, turing_small, turing_flash), evaluator models with the eval stack built in

Proprietary evaluator models, available in the Future AGI platform and through the agent-learning-kit SDK.

Where it earns its bill. Future AGI has custom evals: write your own grading rule as an LLM-as-a-Judge, Agent Evaluator or Code eval, pick the evaluator model, set a pass/fail threshold and run it as a CI gate. Built-in evaluators include Groundedness, Context Adherence, Detect Hallucination, Task Completion, Instruction Adherence, Conversation Resolution and Tone.

The evaluator models are Turing Large (text, image, audio; max accuracy), Turing Small (high fidelity at lower cost) and Turing Flash (fast, low latency), served from Future AGI’s cloud. Image evals run on any of the three, and audio evals go to Turing Large.

Cost. Turing-backed LLM-as-a-judge runs are billed per run in AI Credits (2K credits free each month, then $10 per 1K). BYOK judges cost $0 on the platform and heuristic evals are always free. See pricing.

Latency. Turing Flash is the low-latency tier, Turing Small medium, Turing Large higher.

Self-preference. The bias surface is rubric framing rather than family preference. Calibrate on a human-labeled set per rubric.

Integrations. The Turing models compose natively with traceAI spans, Error Feed clustering, and the optimizers in agent-learning-kit. BYOK any other judge through the same SDK (see the Future AGI section below).

Best for. Production teams running traces, evals, optimization, and gateway routing in one stack. The fast first-pass slot with turing_flash. BYOK plus Future AGI-routed scoring in the same workflow.

2. Claude Sonnet 5.5 and Opus 5.5

Closed-weight frontier. Anthropic API, Bedrock, Google Cloud.

Where it earns its bill. Subjective rubrics that need open-ended reasoning over long context, such as multi-document RAG faithfulness, multi-turn conversation adherence, and agent-trajectory rubrics. Both models have a 1M-token context window, so long production transcripts fit in one pass.

Cost. $2 input / $10 output per 1M tokens for Sonnet 5.5 and $4 / $20 for Opus 5.5. A 2K-input, 200-output call on Sonnet 5.5 costs about $0.006 before the tokenizer effect: Claude 4.7 and later use a tokenizer that produces about 30% more tokens for the same text, so measure real cost per score on your own transcripts. Viable as the second-stage judge in a cascade, expensive as the first pass.

Self-preference. In Zheng et al. 2023, Claude-v1 gave its own answers a 25% higher win rate, though the authors called the evidence inconclusive. Do not use Sonnet 5.5 to judge a Sonnet 5.5 candidate; pair it with GPT or Gemini in an ensemble for launch decisions.

Worth flagging. Claude Sonnet 4.5 retires from the Claude API on November 30, 2026 (replacement: Sonnet 5.5); re-calibrate before you switch.

Best for. High-stakes judging where calibration matters more than cost. Pre-launch validation. The frontier slot in a two-stage cascade.

3. GPT-6.1 Sol and GPT-6 Luna

Closed-weight frontier. OpenAI API.

Where it earns its bill. Structured-output evaluation in OpenAI stacks. At a 5% parse-failure rate, 100K judgments mean 5,000 retries.

OpenAI, Anthropic and Google all ship schema-constrained structured outputs, so measure the parse-failure rate of your rubric on your own labeled set before choosing a judge. In the two-stage cascade, GPT-6 Luna screens and GPT-6.1 Sol rescores.

Cost. Per 1M tokens, standard tier: gpt-6.1-sol $2 input / $10 output, gpt-6-luna $0.10 / $0.50 (20x cheaper than Sol), gpt-6-astra $10 / $50. Batch is half price. The gpt-5-2025-08-07 and gpt-5-mini-2025-08-07 snapshots are scheduled to shut down on December 11, 2026.

Self-preference. In Zheng et al. 2023, GPT-4 gave its own answers a 10% higher win rate, though the authors called the evidence inconclusive. Measure it for GPT-6.1 Sol and GPT-6 Luna on your own set.

Best for. General-purpose judge in OpenAI stacks. Strict structured-output rubrics where parse failures kill throughput.

4. Gemini 4 Argon and Gemini 3.8 Flash for long context and multimodal judging

Closed-weight frontier. Gemini API.

Where it earns its bill. Long inputs and multimodal judging on image, audio, and video inputs (see multimodal LLM-as-a-judge). Gemini 3.8 Flash takes up to 1,048,576 input tokens. Gemini 2.5 Pro and Flash remain available (no shutdown date) if you already calibrated on them.

Cost. Gemini 4 Argon costs $2 input / $10 output per 1M tokens through November 30, 2026, then $4 / $20, and is not on the free tier. Gemini 3.8 Flash costs $0.75 / $3.75 through December 31, 2026, then $1.50 / $7.50.

Self-preference. Zheng et al. 2023 did not test Gemini as a judge. Assume the bias is present, measure it on your set, and do not judge Gemini candidates with a Gemini judge.

Best for. Long-context judging. Multimodal rubrics. Cost-sensitive high-volume scoring with Flash.

5. Luna-2 from Splunk Agent Observability (formerly Galileo)

Closed-weight fine-tune. Splunk Agent Observability (formerly Galileo).

Where it earns its bill. Luna-2 is Galileo’s small language model built for scaling AI evaluations, fine-tuned for low latency and reduced cost on metric evaluations. As of August 7, 2026, Galileo is Splunk Agent Observability, and its plans include Luna tokens (500 million a year on the entry plan, which starts at $100 a month billed annually).

Cost. $0.02 per 1M tokens in the vendor’s own table.

Self-preference. Luna-2 is not a general assistant, so the bias surface is rubric framing rather than family preference.

Best for. Splunk Agent Observability teams that want a purpose-built evaluator.

6. DeepSeek V4.1 Flash, a low-cost open-weight judge

Open weight. MIT license.

Where it earns its bill. Open-weight cost economics. DeepSeek V4.1 Flash weights are MIT: 552B backbone parameters, 8B active per token in prefill and 16B in decode, 1M context.

Cost. Per 1M tokens on the DeepSeek API (peak / off-peak): input cache miss $0.30 / $0.15, output $1.20 / $0.60. A 2K-in/200-out score costs about $0.00084 at peak against about $0.006 on Sonnet 5.5 or GPT-6.1 Sol.

Hosting. Self-hosting 552B parameters takes a multi-GPU node.

Worth flagging. Geopolitical and data-handling considerations apply for some procurement contexts. Calibrate before you scale it.

Best for. High-volume routine scoring where cost-per-judgment binds. The first-pass judge in a two-stage cascade.

7. Llama 3.3 70B for regulated self-hosting

Open weight. Meta’s Llama 3.3 license.

Where it earns its bill. Regulated workloads where data cannot leave the boundary. Llama 3.3 70B is a dense open-weight model you can self-host as a judge. Meta’s Llama 4 Scout and Maverick (April 2025) are mixture-of-experts models.

Cost. Free weights. About $1.04 per 1M input and output tokens on Together’s serverless tier.

Self-preference. Zheng et al. 2023 did not test Llama as a judge; measure self-preference on your own set.

Best for. Regulated workloads (healthcare, banking, defense), on-premise deployments, and cost-optimized scoring where hosted closed models are a non-starter.

8. Prometheus 2 (7B and 8x7B), the open-weight evaluator fine-tune

Open weight. Apache-2.0. arXiv 2405.01535.

Where it earns its bill. Prometheus 2 is fine-tuned from Mistral-Instruct on 100K feedback examples (Feedback Collection) plus 200K pairwise examples (Preference Collection). On four direct-assessment and four pairwise benchmarks it scored the highest correlation and agreement with human and proprietary-judge labels among the open evaluator models tested.

Two variants ship (7B dense and 8x7B MoE), and the training data is subject to OpenAI’s terms for generated data. Pick the 7B for cheap volume and the 8x7B when agreement matters more than GPU budget.

Cost. Free weights under Apache-2.0; you pay only for the GPUs you serve it on.

Self-preference. Measure it on your own set before using it on pairwise launch decisions.

Worth flagging. Fine-tuned judges behave like task-specific classifiers outside their training domain (arXiv 2403.02839).

Best for. Open-source eval stacks. Research workflows where reproducibility matters. The open-weight slot in a cascade where data residency or budget forbids hosted frontier judges.

How do the best LLM judge models compare?

Small and fine-tuned judges carry the volume, frontier judges take the close calls (the cascade below), and open-weight judges are the path for regulated workloads. How close an open judge gets to frontier on your rubric is the number to measure on your labeled set.

What are the best open-source LLM judge models?

Open-source LLM judge models worth testing fall into two groups: models fine-tuned only for grading, and general open-weight models you prompt as judges. Both groups need the same calibration on your labeled set before you trust a score.

• Prometheus 2 (7B and 8x7B, Apache-2.0) is fine-tuned from Mistral-Instruct for direct assessment and pairwise ranking.
• Atla Selene 1 Mini (8B, Apache-2.0) is based on Llama 3.1 8B Instruct with a 128K context; its model card reports it outperforming GPT-4o on RewardBench, EvalBiasBench and AutoJ. A larger Selene 1 is built on Llama 3.3 70B.
• CompassJudger-2 (7B and 32B, Apache-2.0), released by OpenCompass in July 2025, is a Qwen-family generalist judge trained with verifiable rewards. It ships JudgerBenchV2, 10,000 questions across 10 scenarios.
• JudgeLM (arXiv, October 2023) fine-tunes judges at 7B, 13B and 33B parameters.
• General open-weight models such as DeepSeek V4.1 Flash (MIT) and Llama 3.3 70B can be prompted as judges.

Open fine-tuned judge models such as JudgeLM and Prometheus behave like task-specific classifiers outside their training domain (An Empirical Study of LLM-as-a-Judge for LLM Evaluation, arXiv 2403.02839). A 2026 study of 37 LLMs (arXiv 2603.22214) found high correlation with human assessments when paired with a suitable prompt, in particular for GPT-4o and several open models of 32B parameters or more.

Calibrate each on your labeled set; Future AGI takes any of them as a custom evaluator model, including self-hosted endpoints. To run them inside a harness, compare the open-source eval frameworks.

What is meta-judging, and do you need a judge for your judge?

Meta-judging means using an LLM to evaluate other judges’ verdicts. A January 2026 survey, Meta-Judging with Large Language Models: Concepts, Methods, and Challenges (arXiv 2601.17312), maps the methods. We could not verify an open-source model called MetaJudge, so treat the term as a technique, not a product.

In practice, run a second judge from a different model family on the same items, send every disagreement to a human, and fold those labels back into your calibration set. To test the judge itself, hold the content fixed, vary the surface (formatting, verbosity, paraphrase) and measure how far the score moves; the Judge reliability guide runs that perturbation test offline.

The cascade pattern beats picking one judge

Four layers replace the single-judge question. Each layer handles the cases the one below it cannot settle.

Layer 1: deterministic floor. JSON schema, refusal regex, parser. Free, never drifts.

Layer 2: small evaluator judge. turing_flash, Luna-2, or a fine-tuned small open model. Runs on every span. Catches confident pass and confident fail; flags close calls.

Layer 3: frontier general-purpose judge. Sonnet 5.5, GPT-6.1 Sol, Gemini 4 Argon. Runs only on the close calls layer 2 flagged. Bills for reasoning only when reasoning is the binding constraint.

Layer 4: human review queue. Cases the frontier judge still scores ambiguously go to a labeling queue. Labels feed back into rubric calibration so layer 2 catches more next quarter.

A million-span-a-day frontier-only workload is a $300K-to-$1.5M monthly bill. Run the same workload as a cascade that settles 90% of spans at layer 2, and only the remaining 10% reach the frontier judge.

The agent-learning-kit SDK ships the cascade as one call (augment=True runs the local heuristic first and passes its scores and reasoning to the LLM judge you name in model=). The why-LLM-as-a-judge post linked above covers the full pattern.

How to pick: a 90-minute calibration sprint

You do not pick a judge model from a blog post. You pick by running the candidates on your data. This is the cheapest version of that.

1. Hand-label 100 to 300 examples that cover the failure modes your rubric needs to catch. Use two annotators where possible and discard low-agreement items. The set is your ground truth for the next quarter.

2. Run three or four candidate judges from this list against the same set: a frontier closed model (Sonnet 5.5 or GPT-6.1 Sol), a low-cost tier (Gemini 3.8 Flash or GPT-6 Luna), a purpose-built evaluator (turing_large or Luna-2), and an open-weight model (DeepSeek V4.1 Flash or Llama 3.3 70B). Same rubric prompt, structured output, position randomized on pairwise. In Future AGI, bring each candidate in as a custom evaluator model (a supported provider such as OpenAI, AWS Bedrock or Vertex AI, or any HTTP endpoint, including self-hosted open-weight judges) so all of them score the same dataset side by side.

3. Compute Cohen’s kappa or accuracy against the human labels for each judge. Plot kappa on the x-axis and cost-per-score on the y-axis. The Pareto frontier is your shortlist. For the full calibration routine, see LLM-as-judge best practices.

4. Test self-preference explicitly. Take 50 pairs where one side is from the candidate model family and one side is from a different family with human labels for which is actually better. Score in both orderings. The judge whose winrate diverges from the ground truth by more than 10% has self-preference you cannot ignore.

5. Project cost at three months. Token cost times tokens-per-judgment times sample rate times retries. The cheapest judge that hits your kappa threshold wins, not the highest kappa.

The sprint can surface a two-judge cascade (one small evaluator for layer 2, one frontier for layer 3) and one open-weight fallback for the regulated path. That is the procurement output, not “judge model X is best.”

Common mistakes when picking an LLM judge model

• Defaulting to last year’s model. A new judge version can shift behavior. A judge calibrated on GPT-4 is not calibrated on GPT-5 without re-validation. Pin the judge model id inside the eval contract; bump deliberately.
• Keeping a judge past its retirement date. Claude Sonnet 4.5 retires from the Claude API on November 30, 2026, and the GPT-5 snapshots are scheduled to shut down on December 11, 2026. Pin the id, track the vendor deprecation page, and re-run the calibration set before switching.
• Same model as judge and candidate. In Zheng et al. 2023, GPT-4 and Claude-v1 gave their own answers 10% and 25% higher win rates. The cardinal mistake.
• Pricing only the per-token cost. Real cost equals tokens-per-judgment times retries times sample rate. A judge with 5% parsing failures costs more than the per-token rate suggests.
• Trusting a vendor leaderboard number. A vendor benchmark measures the vendor’s tasks, not yours. Calibrate.
• Treating judging as a model choice. Production judging is a workflow: cascade, calibrate, audit, re-calibrate. The model is one input.

What changed for LLM judges in 2025 and 2026?

Running your judge shortlist end to end in Future AGI

The judge model is one component. The stack around it is what compounds across two years of production.

The agent-learning-kit SDK (Apache-2.0) ships CustomLLMJudge, a Jinja2-templated G-Eval primitive that runs against any LiteLLM-backed judge, including every model in this list. The same rubric runs on Turing models or any model you bring. Drop model="turing_flash" for a fast first pass, model="gpt-6.1-sol" or model="claude-sonnet-5-5" for the frontier rescore, or any open-weight model for the self-hosted path.

traceAI carries the same rubric as a span-attached EvalTag in its Python, TypeScript and C# SDKs. The same rubric runs in pytest as a CI gate, in batch on offline traces, and on live spans. That closes the eval-versus-production drift covered in the trace-eval gap post.

The Agent Command Center gateway handles judge routing across 100+ providers with exact and semantic caching, shadow / mirror / race routing, virtual keys with per-key budgets, and PII, prompt-injection, secrets, topic and MCP tool-security guardrails, plus vendor adapters such as Lakera, Presidio and Llama Guard. Canary judge swaps become A/B tests rather than deploy events.

Future AGI is SOC 2 Type II, GDPR, CCPA and ISO 27001 certified and HIPAA compliant (BAA available). On the Future AGI Platform, feedback you record on a wrong verdict is stored against the eval and handed to the evaluator model as few-shot examples on later runs (Feedback), and Error Feed groups similar failed traces into one issue you can work through.

Catching the refund error the 0.91 rubric missed

The support agent from the opening needed three habits. Pin the judge id inside the eval, re-run the labeled set whenever the vendor bumps or retires a model (Sonnet 4.5 retires on November 30, 2026), and route a first pass through Turing Flash with a frontier rescore on close calls, so a refund quoted off by an order of magnitude fails the rubric instead of scoring 0.91.

Ready to pick a judge against your own workload? Start with pip install agent-learning-kit or the evaluation docs, drop three candidates from this list into the same evaluate() call on your labeled set, and pick by kappa per dollar. That is the procurement output that holds for two years.

Related reading

• Evaluating LLM Judge Bias Mitigation (2026)
• LLM Arena as a Judge: Pairwise Comparison Evals (2026)
• LLM Judge Prompt Engineering Guide (2026)
• Best LLM-as-Judge Platforms (2026)
• The 2026 LLM Evaluation Playbook