Agent-as-a-Judge: The Research, and When to Use It

TL;DR

Agent-as-a-judge is an evaluation method where an agentic system grades another agent’s entire trajectory — every step, every tool call — instead of scoring only the final answer. It was introduced by Zhuge et al. in October 2024 (arXiv:2410.10934) and accepted as a poster at ICML 2025.

• The core claim: on the paper’s own DevAI benchmark, evaluating OpenHands, agent-as-a-judge reached 92.07% (gray-box) and 90.44% (black-box) alignment with human consensus, against 70.76% and 60.38% for LLM-as-a-judge.
• The cost claim is against humans, not against LLM judges: $30.58 and 118.43 minutes versus 86.5 human hours costed at roughly $1,297.50.
• Against an LLM judge it is barely more expensive, just far slower: $30.58 versus $29.63 on the same pass, but 118.43 minutes versus 10.99.
• The ablation matters more than the headline: the best configuration used five of the paper’s eight modules and dropped three — search, memory and planning. Memory made judgments worse, not better.
• When to use it: long, branching, tool-heavy trajectories where a wrong intermediate step is expensive. Not for single-turn Q&A.
• Watch the name: vendors now ship two different things under “agent-as-a-judge” — one reads traces, one just scores a final output.

Why AI Agents Broke the Old Way of Grading AI

For years, evaluating an LLM meant grading a single input and a single output. You sent a prompt, got a response, and scored it. That worked fine when the system in front of you only ever did one thing at a time.

Agents don’t work that way. They plan, call tools, read the results, call more tools, and only then produce a final answer. A support agent might check an order status, look up a refund policy, call a refund API, and reply to the customer, four or five separate decisions inside one “turn.”

Here’s the problem: a final answer can be correct while everything that led to it was wrong. An agent can call the wrong tool twice, retry a failed action without noticing, and still stumble into a reply that reads fine. Output-only grading has no way to see any of that.

The stakes go up as agents get more autonomy. A chatbot that gives a slightly off answer is annoying. An agent that books a wrong flight, sends a payment to the wrong account, or deletes a file it shouldn’t have touched is a different category of problem, and none of those failures necessarily show up in the final message it sends you.

Where Agent-as-a-Judge Came From

Agent-as-a-judge is itself an agentic system, one that can navigate a trace, call tools and retrieve evidence, built to evaluate the output and the process of another agent. Instead of one LLM call scoring a final response, an agent-judge can read a trace, call tools to check claims against a real source of truth, and revisit earlier steps before it scores anything.

The term traces back to a specific paper: “Agent-as-a-Judge: Evaluate Agents with Agents” (arXiv:2410.10934), submitted 14 October 2024 by a thirteen-author team across KAUST and Meta AI — led by Mingchen Zhuge, with Changsheng Zhao, Dylan Ashley, Yuandong Tian, Vikas Chandra and Jürgen Schmidhuber among the co-authors. It was later accepted as a poster at ICML 2025.

The paper’s own framing is careful, and worth repeating because vendors tend to drop it: agent-as-a-judge is “an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process.” An extension. Not a replacement.

To test the idea, the authors built DevAI, a benchmark of 55 realistic AI development tasks annotated with 365 hierarchical user requirements. Existing coding benchmarks mostly graded whether the final code worked; DevAI was built specifically because that approach misses everything that happens on the way there.

What Did the Agent-as-a-Judge Paper Actually Measure?

Two separate results get collapsed into one talking point everywhere else. They are not the same comparison, and the distinction changes what you should expect from the method.

Result one — alignment, measured against LLM-as-a-judge. Evaluating OpenHands on DevAI, agent-as-a-judge reached 92.07% alignment with human consensus in the gray-box setting and 90.44% in black-box, against 70.76% and 60.38% for LLM-as-a-judge on the same runs.

Result two — cost and time, measured against human evaluation. Three expert evaluators spent a self-reported 86.5 hours on a full DevAI pass, which the authors cost at roughly $1,297.50 assuming a $15/hour rate. Agent-as-a-judge spent $30.58 in API calls and 118.43 minutes, which works out to 2.36% of the cost and 2.28% of the time.

That second number is the one most write-ups misattribute. The saving is against people, not against LLM-as-a-judge.

The cost gap between the two AI judges is the genuine surprise in that section, and it runs against the obvious intuition. On the same DevAI pass, LLM-as-a-judge cost $29.63 against agent-as-a-judge’s $30.58 — about 3% apart. The authors explain why: the single-call judge lacks “intelligent context selection by the Agent-as-a-Judge’s modules,” so it stuffs far more tokens into one call and burns nearly the same spend.

What agent-as-a-judge actually costs is time. It took 118.43 minutes against 10.99 for LLM-as-a-judge, roughly eleven times longer. Budget the latency, not the token bill — and note this is one benchmark, not a general ratio.

The authors benchmarked three open-source coding agents on the same 55 DevAI tasks: MetaGPT, GPT-Pilot, and OpenHands. That the comparison ran against systems teams were actually shipping, rather than a toy setup, is the reason the numbers are worth quoting at all.

They are still single-benchmark, single-domain numbers from the method’s own authors, on code generation specifically. No one has reproduced them on a customer-support or browser agent. Quote them as “what the paper found,” not as a property of the technique.

LLM-as-a-Judge vs. Agent-as-a-Judge

What LLM-as-a-Judge Actually Checks

LLM-as-a-judge grades one input/output pair per call. It never sees the steps in between, so it can’t tell an efficient path from a lucky one, and it can’t tell a correct claim from a confident-sounding one that happens to be false.

It still works well in plenty of places: single-turn Q&A, RAG answer quality, and tone or safety checks on a final piece of text. None of those require watching a process unfold, so a single-call judge is the right amount of machinery for them.

It’s also cheap and fast, which matters more than it sounds. A single-call judge can run on every production response without meaningfully denting latency or spend, which is exactly why it stays the right default for high-volume, low-stakes text generation even after a team adopts trajectory evaluation elsewhere.

What Changes When the Judge Is Also an Agent

An agent-judge can read the full trace of a run, call tools to verify claims against ground truth like a database, a repository, or a live API, and revisit earlier steps before deciding on a score. That’s a fundamentally different unit of evaluation than “does this last message sound right.”

A concrete failure shows the gap. An agent claims in its final reply that it filed a support ticket. LLM-as-a-judge grades that claim exactly as stated, because the sentence reads fine. An agent-as-a-judge checks the ticketing system directly and catches that no ticket was actually created.

That extra capability isn’t free, though the bill lands where you might not expect it. Reading a full trace, calling verification tools and re-scoring earlier steps means many more round trips per evaluation, so the wall-clock cost climbs even where token spend does not. Treat agent-as-a-judge as a targeted tool for the runs that need it, not a blanket replacement for every eval a team already runs.

What This Post Does Not Cover

This page is about the agent-as-a-judge method specifically: where it came from, what it measured, and when the extra cost pays. Four neighbouring pages own the ground around it, and going there is a better use of your time than reading a thinner version here.

• Designing and calibrating a single-call judge belongs to LLM-as-a-Judge in 2026 — G-Eval, pairwise and rubric methods, Cohen’s kappa calibration, and the four biases a production judge has to control.
• Picking the model that does the judging belongs to Best LLM Judge Models in 2026, which ranks eight models on human correlation, cost per score, latency and self-preference bias.
• The full scoring surface for agents belongs to The Definitive Guide to AI Agent Evaluation — six dimensions, per-dimension CI thresholds, and the 4-D trajectory score. Agent-as-a-judge is one way to produce those scores, not the whole practice.
• The one-paragraph definition, if that’s all you needed, lives in the glossary entry for agent-as-judge.

The Case for Judging the Whole Trajectory, Not Just the Answer

Where Agents Actually Fail

Multi-step agents fail in ways a final-answer check simply can’t see. Four patterns come up repeatedly: redundant or looping tool calls that waste time and money, the wrong tool getting picked for a step that still limps to a workable answer, unsafe intermediate actions that get undone before the final reply, and user constraints that get silently dropped somewhere mid-plan.

No single published failure-rate statistic captures how often each of these happens across agent types and domains, so treat them as documented failure classes rather than reaching for a specific percentage. What matters operationally is that all four are invisible to a judge that only ever sees the last message.

Take the looping case as an example. An agent calls a search tool, gets a partial result, calls the exact same search again because it didn’t recognize the result as final, and repeats that twice more before moving on. The eventual answer might still be correct. The run cost three or four times what it should have, and nothing about the final reply hints at that.

What Trajectory-Level Judging Adds

Trajectory-level judging splits “did it work” into separate, measurable dimensions: step efficiency, tool selection accuracy, and task completion, instead of collapsing everything into one pass or fail. That separation is what lets a team fix the actual broken step instead of guessing.

Output and trajectory checks are complementary, not competing options. LLM-as-a-judge still covers output quality well; agent or trajectory judges cover process quality that output checks were never built to see.

How Does an Agent-as-a-Judge Actually Work?

The Eight Modules in the Original Paper

The paper doesn’t describe a monolithic judge. It builds one out of eight modules: graph, locate, read, search, retrieve, ask, memory, and planning. Graph reconstructs the project structure, locate finds the relevant artifact, read and retrieve pull the evidence, ask produces the verdict on a requirement.

The ablation is more interesting than the headline. The authors report that “the modular combination of (1), (2), (3), (5), and (6) achieved the highest performance” — graph, locate, read, retrieve and ask. Five of the eight, which means three were dropped: search, memory and planning. Search went first: the paper found DevAI’s small workspaces “cannot fully benefit from the search module”. Planning went too, because “the procedure is unstable”.

Memory didn’t just fail to help; it actively hurt. Carrying prior judgments forward meant “any errors in previous judgments could lead to a chain of errors.” A judge that remembers its own mistakes compounds them.

That finding is worth holding onto, because the direction of travel in the field points the other way. A January 2026 survey of agentic judges (arXiv:2601.05111, You et al.) organises the space around four capabilities: planning, tool-augmented verification, multi-agent collaboration, and persistent memory. Two of those four are exactly what the original paper’s ablation removed. Neither position is settled design — treat both as open questions when you build.

What a Production Agent-Judge Adds

Research judges run over a finished benchmark. Production judges run over live traffic, which adds three requirements the paper didn’t have to solve for: trajectories have to be queryable rather than read whole, claims have to be checkable against the systems the agent actually touched, and rubrics have to keep pace with an agent that keeps changing.

Each of those three carries a real cost. Querying a trajectory instead of reading it whole means the judge needs its own notion of relevance, because long-horizon traces run past what fits in a context window. Environment verification means the judge needs credentialed access to the same systems the agent under review touches, which is a permissions decision, not just an engineering one. Adaptive rubrics mean the scoring criteria can’t be frozen the day they’re written, or they drift out of sync with what the agent has evolved into doing.

A Worked Walkthrough

Picture a customer-support agent trace with six steps: read the ticket, look up the order, check refund eligibility, call the refund tool, draft a reply, send it. An LLM-as-a-judge only ever sees step six, the final reply, and scores whether it sounds accurate and polite.

An agent-as-a-judge walks the whole trace. It checks whether the refund tool was actually called with the right order ID at step four, whether the eligibility check at step three matched the stated policy, and only then whether the final message accurately reflects what happened upstream. This example is illustrative, not a documented case study, but it maps directly onto the kind of trace any tool-using support agent produces.

Say the eligibility check at step three was wrong — the customer wasn’t actually eligible for a refund under policy, but the agent approved it anyway and wrote a friendly, well-formatted reply confirming it. An LLM-as-a-judge reading only the final message would likely score that reply highly: it’s polite, clear, and directly answers the customer. The trajectory-level check is the only one that catches the policy violation sitting underneath a fluent response.

Two Different Things Now Ship Under the Same Name

“Agent-as-a-judge” has already split into two meanings, and the gap between them is the difference between trajectory coverage and no trajectory coverage at all.

The first is the paper’s idea built as intended. The judge reads the captured trace for a task, inspects the steps inside it, and writes its scores back onto the spans those steps belong to. What it grades is the process.

The second takes a single input and output pair and scores it against a criteria string, usually as a pass/fail or a number on a fixed scale. That is a genuinely useful check, and it is often an LLM-as-a-judge in everything but the label. The trajectory never enters the evaluation at all.

Both are legitimate checks. But when a tool is described to you as agent-as-a-judge, the question worth asking is which of the two you are actually getting: does the judge read the trace, or does it read the last message? Only the first catches the refund example above.

Future AGI sits on the first definition: eval scores land on the trace spans they belong to, so a trajectory score points at the step that produced it rather than at the run as a whole — the trace-native evaluation pattern.

When to Use Agent-as-a-Judge (and When Not To)

Signals You Need It

Reach for agent-as-a-judge when trajectories are long or branching, when the agent leans heavily on tools, when a wrong path is costly even if the final answer looks fine, or when the agent acts on live external systems where a bad intermediate step has real consequences. The refund example above covers that last case directly: a wrong intermediate decision can sit underneath a reply that reads as correct, and only a trajectory-level check would have caught it before it shipped.

Signals It’s Overkill

It’s usually unnecessary for simple single-turn tasks, low-stakes internal tools where a wrong answer just gets corrected next turn, and early prototyping, where speed of iteration matters more than trace-level rigor. Adding trajectory judging too early just slows down a team that’s still figuring out what the agent should do at all.

A reasonable rule of thumb: if you can’t picture a scenario where a correct-looking final answer hides a real problem, you probably don’t need trajectory-level judging yet. Add it once that scenario becomes concrete, not before, because building rigor around a problem that hasn’t shown up yet mostly just adds latency and cost to every eval run.

Running Trajectory Checks With Future AGI

You don’t need an agentic judge to start scoring trajectories, and the cheapest layer costs nothing to run. Future AGI’s open-source Agent Learning Kit (pip install ai-evaluation) ships 72 local metrics that run on your machine with zero API calls. Five of them score trajectories directly — task_completion, tool_selection_accuracy, step_efficiency, trajectory_score and reasoning_quality — and action_safety and groundedness cover two more rows of the table above.

Here is the looping failure from earlier, scored locally. No API key, no network call:

from fi.evals.local.registry import get_registry
from fi.evals.metrics.agents.types import (
    AgentTrajectoryInput, AgentStep, ToolCall, TaskDefinition,
)

# A support agent that searched three times for the same thing, then refunded.
trajectory = [
    AgentStep(step_number=1, tool_calls=[ToolCall(name="search_orders", arguments={"q": "order 8812"})]),
    AgentStep(step_number=2, tool_calls=[ToolCall(name="search_orders", arguments={"q": "order 8812"})]),
    AgentStep(step_number=3, tool_calls=[ToolCall(name="search_orders", arguments={"q": "order 8812"})]),
    AgentStep(step_number=4, tool_calls=[ToolCall(name="issue_refund", arguments={"order_id": "8812"})],
              is_final=True, observation="Refund issued."),
]

payload = AgentTrajectoryInput(
    task=TaskDefinition(
        description="Refund order 8812",
        required_tools=["search_orders", "issue_refund"],
        max_steps=2,
    ),
    trajectory=trajectory,
)

registry = get_registry()
for name in ["task_completion", "tool_selection_accuracy", "step_efficiency"]:
    result = registry.get(name)().compute_one(payload)
    print(f"{name:26} {result['output']:.2f}  {result['reason']}")

Output, on ai-evaluation 1.1.0:

task_completion            0.40  Agent reached final step
tool_selection_accuracy    1.00  Required tools: 2/2 used. Success rate: 100%
step_efficiency            0.65  4 steps taken, 2 redundant

Tool selection scores a perfect 1.00 — the agent picked the right tools and every call succeeded. The refund went through. An output-only judge would have passed this run without comment. step_efficiency is the metric that names the actual problem: 4 steps taken, 2 redundant.

That is the whole argument of this post, running for free on a laptop. Deterministic trajectory metrics catch a large share of process failures before you spend anything on an agentic judge — which is the point at which the Evaluate platform is worth reaching for, with 50+ evals, LLM-as-judge, and custom evals you define with a grading rule, a dataset column mapping and a pass/fail threshold, callable from your own CI. Escalation is a flag: evaluate(..., augment=True) runs the local metric first and only sends the hard cases to a judge.

None of this works without a trace to score. Future AGI’s Observe surface provides OpenTelemetry-native tracing across 30+ agent frameworks, which is what any trajectory-level judge reads from in the first place — a judge can only walk a trace that was actually captured. The instrumentation layer is traceAI, an open-source OTel library, so you can sign up free and point it at a hosted project, or self-host and keep every trace inside your own environment. For what a well-instrumented trace should capture beyond the model call, see prompt, context, harness and loop.

Keeping evaluation and tracing in one place is what makes that loop short: the trace you capture is the trace you score, rather than two tools on two schedules with an export step in between. For a wider view of how to pick an approach, see the agent evaluation frameworks comparison and evaluating browser-use agents, which covers a class of trajectory failure specific to agents that operate a browser.

The practical starting point is small: pick one agent workflow with real tool calls, capture its traces, and run the three local metrics above on the step your team already suspects is unreliable. That’s a narrower, cheaper version of the DevAI methodology applied to one workflow, and it’s usually enough to tell you whether an agentic judge earns its cost for that particular agent.

Conclusion

Agents fail in the steps, not just in the final answer, and grading only the answer misses exactly that. Agent-as-a-judge is a research-backed method for closing that gap, not a rebrand of LLM-as-a-judge — Zhuge et al.’s 2024 paper and its ICML 2025 acceptance gave the idea a benchmark, a methodology and published numbers.

Read those numbers carefully. The alignment gain is measured against LLM-as-a-judge; the cost saving is measured against human evaluators. And the paper’s own ablation dropped three of its eight modules — search, memory and planning — from the winning configuration, so the “more agentic is better” instinct is not what the evidence supports yet.

The practical path is layered. Start with deterministic trajectory metrics, which are free and catch redundant calls and wrong tool selection on their own. Add output-level LLM-as-a-judge for quality. Escalate to an agentic judge only for the workflows where a correct-looking answer can hide an expensive wrong step.

Neither output checks nor trajectory checks replace the other. They answer different questions, and a mature evaluation practice runs both, pointed at the parts of the system each one is actually built to catch.