Agent Eval Harness: How to Evaluate AI Agents, Not Just Models

A single pass-or-fail on the final answer misses what an agent actually did. Agents are judged on the whole path they take: how trajectory grading, tool-call checks, and environment outcomes come together.

You have an agent that plans, calls tools, and works across many steps toward a goal. Scoring it the way you score a model, one prompt in and one answer out, throws away almost everything that matters. The tool calls it made, the states it passed through, and the path it chose are the actual behavior you care about.

An agent eval harness scores that whole path. This guide covers what it measures, the two ways to grade a run, the four parts every agent evaluation has, the three grader types, the split between pass@k and pass^k, and the benchmarks that power the field.

If you want the model-level version first, what an eval harness is explains how the software around a benchmark turns it into a reproducible score. This post is the agent-level counterpart: the same idea, with many more moving parts to measure.

The Direct Answer: What an Agent Eval Harness Measures

An agent eval harness runs an agent through tasks in an environment and scores the whole trajectory: the sequence of tool calls, the intermediate states, and the final outcome, not a single output string. It measures behavior across many steps, so a run can fail even when the last message reads well.

A model eval asks one question and checks one answer. An agent eval sets a goal, hands the agent an environment such as a code repository, a browser, or an operating system, and lets it act. The harness records everything it does and then judges the result against the task’s success condition.

The unit of measurement changes from a string to a trajectory. The harness asks whether the agent called the right tools, in a sensible order, and left the environment in the correct end state. That shift is why the same model can top a static benchmark and still stumble as an agent, where the score depends on a chain of decisions instead of one reply.

The environment is the source of truth. It starts from a known state, the agent acts on it through tools, and the harness reads the final state back to decide success. Because each trial resets that environment to the same starting point, one agent can be run hundreds of times and scored identically, which is what makes a messy, multi-step process measurable at all.

Change that starting state and the same agent can score differently, so the environment definition is part of the test, not a detail you set once and forget.

Because behavior spans steps, an agent harness evaluation leans on repetition and on grading logic that can read a whole run. The next sections name those pieces, starting with the two angles you can grade a run from.

Transcripts vs Outcomes: Two Ways to Grade an Agent

Anthropic frames two grading modes. Outcome grading asks whether the final environment state is correct. Transcript grading asks whether the steps along the way were sensible. Outcome grading is objective but coarse; transcript grading catches lucky or wasteful paths that still landed on the right answer.

The outcome is the final state in the environment at the end of a run. If the task was to fix a failing test, outcome grading runs the suite and checks that it passes. It does not inspect how the agent got there, which makes it a clean, automatable signal of success or failure.

A transcript, also called a trace or trajectory, is the complete record of a run: the outputs, tool calls, reasoning, and intermediate results. Transcript grading reads that record and asks whether the agent behaved well, for example whether it opened the right files, avoided a destructive command, and stopped once the job was done.

A single task shows why both matter. Told to fix a failing test, one agent edits the offending function and the suite goes green; another comments the test out and the suite also goes green. Outcome grading passes both, since the final state looks correct either way. Transcript grading separates them, because the record shows one agent solved the problem and the other hid it.

The two modes do different jobs, and Anthropic’s guide to agent evals treats them as complementary. Use outcome grading to decide pass or fail at scale, and reach for transcript grading when you need to know why a run failed or whether a pass was luck.

The Four Components of an Agent Eval

Every agent evaluation breaks into four parts. Anthropic names them the task, a single test with defined inputs and success criteria; the trial, one attempt at that task; the grader, the logic that scores a run; and the evaluation harness, the infrastructure that runs the whole thing end to end.

A task is one problem with a clear pass condition. A trial is one attempt at it, and you run many trials per task because agents are non-deterministic and a single run tells you little. A grader scores each trial. The evaluation harness ties them together: it loads tasks, launches trials, collects the transcripts and outcomes, and applies the graders.

One distinction trips people up. The evaluation harness is not the agent harness. The agent harness is what runs the agent: its loop, its tools, and its memory. The evaluation harness is what runs the measurement around it. Blurring the two is exactly why a coding agent harness benchmark number can move when only the harness changed and the model stayed fixed.

That infrastructure is also what makes results portable. When the evaluation harness pins the task set, the trial count, and the graders, another team can rerun the same evaluation and get comparable numbers. Naming these parts is what turns a one-off score into something reproducible, which is the whole reason to build a harness instead of scoring runs by hand.

So hold them apart. The agent harness produces the behavior, and the evaluation harness measures it. Once the four parts are named, the interesting choice is the grader, because how you score a run decides what the whole evaluation actually rewards.

The Three Grader Types

Graders come in three kinds. Code-based graders run rule or assertion checks on the final state. Model-based graders use an LLM as a judge. Human graders have people rate runs. Each trades cost, scale, and subjectivity differently, so most serious evaluations combine more than one.

Code-based grading is the fastest and cheapest, and it is fully objective when the task has a checkable answer: a test passes, a file exists, an API returns the expected value. Its limit is rigidity, since it only checks what you wrote a rule for and misses anything you did not anticipate.

Model-based grading, or LLM-as-judge, scales to open-ended outputs that no rule captures cleanly, such as whether a summary is faithful or a reply is helpful. The catch is calibration: the judge model needs its own checking against human labels, or its scores drift away from what you actually mean by good.

Human grading gives the highest fidelity for taste and nuance, and it is the slowest and most expensive to run. The usual pattern is to reserve it for the trials where code and model graders disagree, and for periodic audits that keep the automated graders honest.

In practice the three stack. A code-based grader gates the objective part of a task, a model-based grader scores the open-ended part, and humans spot-check a sample of both. Layering them keeps cost down while covering the parts of a run that no single grader can judge on its own, which is why production evaluations rarely rely on one type alone.

pass@k vs pass^k: Capability vs Consistency

Two metrics answer different questions. pass@k counts a task solved if any of k attempts succeeds, so it measures peak capability and rises as k grows. pass^k counts it solved only if all k attempts succeed, so it measures consistency and falls as k grows. Deployed agents need the second as much as the first.

pass@k comes from code generation, where the Codex evaluation counted a problem solved if any sampled attempt passed the unit tests. It rewards an agent that can reach the answer eventually, which is the right question when you are probing how capable a model is at its best.

pass^k comes from tau-bench, which introduced it to measure reliability across repeated trials. Demanding that every attempt succeed is a harsh bar, and it surfaces agents that pass on average yet fail unpredictably. tau-bench’s own result drives the point home: its strongest agent passed well over half of retail tasks on a single try, but asked to pass the same task eight times in a row, its success rate fell below 25%.

The practical habit is to report which metric you used and the value of k. A pass@5 and a pass^5 on the same agent describe two different things, and an agent that looks production-ready on one can look shaky on the other. Capability tells you the ceiling; consistency tells you what a user will actually get.

The number k is a budget, not a footnote. Reporting pass@1 next to pass@10 shows how much retrying buys you, and pass^k at a few values of k shows how quickly reliability decays as the bar rises. A leaderboard that hides k is hard to trust, because the same agent can post very different figures depending on how many attempts it was quietly allowed.

The Benchmarks That Power Agent Evaluation

The field’s standard environments span domains: SWE-bench for code, WebArena for the web, OSWorld for desktop control, tau-bench for tool use, and GAIA for general assistance. Princeton HAL runs many of them under one standardized harness so scores line up across agents.

Each benchmark is an environment plus a task set. SWE-bench hands an agent a real GitHub issue and its repository and checks whether the agent’s patch makes the tests pass; the Verified subset of 500 problems, human-filtered by OpenAI, is the slice most scores quote, drawn from a full set of about 2,294.

The others cover different surfaces. WebArena gives 812 tasks inside self-hosted web apps, and OSWorld sets 369 tasks across real desktop software. tau-bench scripts customer-service conversations in retail and airline domains, and GAIA poses 466 general-assistant questions that need tools and reasoning to answer.

Comparing results across these environments is the hard part, which is what Princeton HAL exists to fix. The Holistic Agent Leaderboard runs agents through many benchmarks under one cost-controlled harness. Its paper reports 21,730 rollouts across 9 models and 9 benchmarks for about $40,000, a snapshot of how expensive thorough agent evaluation gets. The repository is archived, so treat those figures as a point-in-time reference.

Public benchmarks are a starting point, not the finish line. If your agent does something none of them covers, the same machinery applies to a task set you write yourself: define tasks with clear success conditions, wrap them in an environment, and score trials with a grader. A handful of well-chosen goldens from your own domain often predicts production behavior better than a rank on a leaderboard built for someone else’s tasks.

To evaluate your own agent, match the environment to what it does, then run enough trials to separate skill from luck, pick graders that fit the task, and report pass@k or pass^k with the value of k. If code is your domain, running SWE-bench in practice is the next step. The agent you started with stops being a demo and becomes something you can measure the same way every time.