How to Build an AI Agent Benchmarking Suite From Real User Tasks

Your team upgrades to the model at the top of a public agent leaderboard. A week later, support tickets show the agent now fails at generating return labels, a task many of your users bring every day, and the leaderboard never contained it.

The fix is a test built from the work your agent does every day, starting with the tasks users complain about first.

A private benchmark built from real user tasks can then decide every agent change: source the tasks, define success by end state, pin the set, compare configurations with an interval, and wire the result into a release gate.

TL;DR

• Public benchmarks measure someone else’s tasks, so build a private suite from your own sessions, confirmed failures, and simulated scenarios.
• Define success by the end state a task should leave behind, plus required and forbidden tool calls and a budget.
• Version the task set and pin every run to a version, so results compare like with like.
• Compare configurations task by task with a confidence interval, and ship a cheaper change only if the worst case stays within an agreed margin.
• Sandbox risky tasks, separate infrastructure failures from quality failures, and gate releases on the result.

Why public AI agent benchmarks can’t rank your agent on your users’ tasks

Public AI agent benchmarks use their own tasks, tools, and data. Your agent calls your APIs and answers in your users’ phrasing.

Scaffolding differs too: the prompt, the tool definitions, and the retry logic around the model can move scores and rankings enough to reorder a leaderboard, a point Kapoor and colleagues make about agent leaderboards. Our post on benchmarks vs production evals covers the gap in depth.

Public sets also age. Once a benchmark is popular, its tasks can leak into training data, and OpenAI stopped reporting SWE-bench Verified after finding frontier models could reproduce its reference fixes. Top scores also bunch near the ceiling, which Anthropic notes for SWE-bench Verified at over 80%.

Some graders check too little. One popular benchmark scored a do-nothing agent correct on 38% of airline tasks because its substring and database-state checks could not tell an unchanged state from a correct one.

Read those numbers with care, as our guide to reading coding agent benchmark scores explains. Treat a leaderboard as a filter for which models to try. Then let your own suite, the one holding the return-label task, decide which one ships.

Source benchmark tasks from real user sessions

A useful private suite mixes three sources, each covering a gap the others leave. Choose the mix so the suite reflects how your users work and where your agent breaks.

Sessions from production traces

Sample real sessions from your traces, grouped by user intent and by outcome. Sampling at random alone over-represents the easy, frequent tasks. Make sure every common intent appears, including the dull ones like return labels, since those are the tasks users notice first when they break.

Failures confirmed by reviewers

Pull failed sessions and have a reviewer confirm each one is a real failure. The reviewer also writes the expected outcome, which becomes the task’s success check. You can export reviewed cases to a dataset so confirmed failures flow into the suite without copy and paste.

Simulated scenarios for what production has not shown yet

New features and rare but costly paths have no production history yet. Simulation fills that gap. You can also rebuild a production session as a scenario and vary it. Our guides on golden set design and AI agent simulation cover both techniques.

Before any session enters the suite, strip personal data and replace it with realistic stand-ins. Keep the structure of the request, such as an order in an odd state, and drop the identity of the person behind it. Clean data lets every engineer run the suite locally.

Define success per benchmark task by its end state

Each task needs a pass condition that checks what happened in the world. For a return-label task, the end state is that a label exists for the right order. For an account change, the record changed and nothing else did. A wording-only check would let a failed action pass.

Tool use adds two more checks. Name the minimum tools a task must call, such as create_label, and the ones it must never call, such as a refund tool on an information-only request. Grade the outcome rather than the exact order of calls, since several paths can be valid.

Then set a cost and latency budget per task, so a configuration that passes by burning tokens still shows up as worse.

Use a judge only where meaning matters, such as whether an explanation to the user was accurate and polite. Every other check should be code that reads the end state or the trace. Code checks give the same answer every run, which keeps the suite stable from one release to the next.

Version the task set and pin every run to it

A benchmark score only means something if the tasks stay fixed between runs. Version the task set and write change notes for every version. Changes to a task go into a new version, so last month’s results still point to the exact tasks they ran against.

Each run should record the task-set version, the agent version, the model, the grader or judge version, and the seed or temperature. Pinning the agent version per run means a rerun on the same scenarios isolates what you changed. Without that record, a grader update can move a score when nothing shipped.

Retire tasks through the same versioning. When a feature is removed or a tool changes shape, mark the old tasks as retired in a new version, with a note on why. The history stays readable, and a pass-rate jump can always be traced to the tasks that changed.

How to benchmark AI agents: one pinned set, many configurations

To benchmark AI agents fairly, run every prompt, agent, and model configuration against the same pinned task set. Agents are not deterministic, so repeat each run and record the task-set version it used.

You can run configurations over one dataset and choose a winner on eval scores, response time, and tokens together.

Compare results task by task with a confidence interval around the pass rate, so you can tell a real gain from noise. The code below computes a paired bootstrap interval for the change in pass rate, then turns it into a gate decision.

import random
import statistics

def paired_bootstrap(baseline, candidate, resamples=10000, seed=7):
    """95% interval for the pass-rate change. Both configs ran the same pinned tasks, so resample task indices."""
    assert len(baseline) == len(candidate), "results must be aligned task by task"
    rng = random.Random(seed)
    n = len(baseline)
    diffs = []
    for _ in range(resamples):
        idx = rng.choices(range(n), k=n)
        diffs.append(statistics.fmean(candidate[i] - baseline[i] for i in idx))
    cuts = statistics.quantiles(diffs, n=40)
    return statistics.fmean(c - b for b, c in zip(baseline, candidate)), cuts[0], cuts[-1]

def release_gate(baseline, candidate, cheaper=False, margin=0.03):
    """Block a clear loss, ship a clear win, and ship a cheaper change (fewer tokens or lower latency) only if the worst case stays within the margin."""
    delta, low, high = paired_bootstrap(baseline, candidate)
    if high < 0:
        return "block", delta, (low, high)
    if low > 0:
        return "ship: clear win", delta, (low, high)
    if cheaper and low > -margin:
        return "ship: cheaper, and no worse than the margin", delta, (low, high)
    return "keep current: inconclusive", delta, (low, high)

# 50 pinned tasks, results aligned by task: both pass 34, baseline only 2, candidate only 8, both fail 6.
# With repeated runs, pass each task's mean pass rate (for example 2 of 3 runs = 0.67) instead of 0 or 1.
baseline  = [1] * 34 + [1] * 2 + [0] * 8 + [0] * 6    # 36 of 50
candidate = [1] * 34 + [0] * 2 + [1] * 8 + [0] * 6    # 42 of 50
verdict, delta, ci = release_gate(baseline, candidate)
print(verdict, round(delta, 2), [round(x, 2) for x in ci])   # keep current: inconclusive 0.12 [0.0, 0.24]

Both configurations ran the same 50 pinned tasks, so the code resamples task indices and compares results task by task. The candidate passes 42 tasks against the baseline’s 36, a gain of 0.12, with a 95% interval from 0.0 to 0.24. The interval includes zero, so the gate calls it inconclusive and keeps the current version.

quantiles(n=40) splits the 10,000 resampled changes into 40 equal slices, so the first and last cut points (2.5% and 97.5%) bound the 95% interval. Pairing by task narrows it here: treating the runs as unrelated samples gives -0.04 to 0.28.

A McNemar test, which looks only at the 10 tasks that flipped (8 toward the candidate, 2 toward the baseline), gives p = 0.11, the same inconclusive verdict. Our post on significance tests for LLM evals walks through it.

With only ten tasks that changed, the lower bound sits right on zero, so use at least 10,000 resamples or it can flip with the seed. Repeated runs go in as per-task averages, never as extra rows, or the interval comes out falsely narrow.

With 50 tasks, the interval runs about 12 points either side of the measured change, so smaller gains stay inconclusive. To catch a real 5-point change 80% of the time, plan on roughly 300 to 600 tasks, assuming 10 to 20% of tasks flip between versions.

Repeats reduce noise within a task, but more tasks is what narrows the interval most.

Sandbox coding, RL, and voice tasks, and watch for reward hacking

Some tasks run code or act in an environment, and they need isolation. Give each task a fresh sandbox, turn the network off for untrusted code, and keep the grader where the agent cannot read or edit it. Otherwise one task can leave state behind that changes the next task’s result.

Agents can also game the objective. A coding agent might edit the tests instead of fixing the bug, or hard-code the expected output. The docs for agent-learn bench sandboxes show one way to isolate coding tasks.

The signals are usually visible in the run. Diff the files the agent touched against the files the task allowed, and flag any change to tests or graders. Compare outputs against the expected values, and flag outputs that match exactly while the logic never ran.

Add canary values that only a gamed run would touch, and review every flagged run by hand before it counts as a pass.

Separate infrastructure breakage from quality failures before trusting a score

A failed task does not always mean a worse agent. Timeouts, rate limits, sandbox crashes, and missing fixtures all fail tasks without saying anything about quality. Mixing them into the score hides real regressions behind noise, and it can make a lucky run look better than the agent is.

Tag every failed task as infrastructure or quality before you compute the score, and classify by cause rather than symptom. A provider 429 or a sandbox crash is infrastructure, while a timeout caused by the agent looping is a quality failure.

Re-run the infrastructure failures, then report both numbers side by side. If infrastructure failures exceed an agreed share of the run, rerun the whole suite before anyone reads the quality score.

Say a run drops from 42 passes to 37, and four of the five new failures are rate-limit errors from a payment sandbox. Rerun those four and the quality picture may not have changed at all. Recording the failure type on every task makes that check take minutes.

Make AI agent benchmarking a release gate

Run the pinned suite in CI on every prompt, tool, or model change, and turn the interval into a decision. Our post on building an agent eval harness shows how to wire the runner into your pipeline.

Agree on the decision rules before the first run, so nobody argues about a result after seeing it. The table below gives a starting set. The margin rule matters most: without it, a cheaper change that lost several points of quality would still ship.

Keep every gate result, and chart the pass rate per task-set version over time. A slow slide across several releases, each one inside the noise, is easy to miss one run at a time, but obvious on the trend line. When the pass rate nears 100%, add harder tasks, because a saturated suite stops telling configurations apart.

Building a private benchmark suite with Future AGI

Replay turns production sessions into scenarios that keep running on every change, so a replayed failure becomes a regression test. Each simulation run pins an agent version, and annotation queues export reviewer-confirmed cases into a dataset for the suite.

Experiments run prompt, agent, and model configurations over that dataset, scored by evals, and the Choose winner step weighs quality against latency and tokens.

The Apache-2.0 agent-learning-kit adds agent-learn bench for coding, RL, and voice lanes, with a subprocess sandbox by default and an opt-in Docker sandbox that turns the network off for untrusted code. Reward-hack detection is on by default for harness-driven runs.

The platform code sits in the Future AGI repo.

Benchmark your own users’ tasks before the next model release

Back to the return-label task. Add it to the pinned suite with an end-state check that a label exists for the right order. When the next leaderboard model arrives, it runs against the same task set as the current one, and the gate decides with an interval and a margin.

Start with the ten sessions your users bring most often, and the suite grows from there.

Replay a production session into your first benchmark scenario today.