Guides

Evaluating Coding Agents in 2026: A Five-Dimension Eval for Your Codebase

Coding agent evaluation for your own repo: replay merged PRs and score tool calls, multi-file coherence, plan, and rollback instead of trusting SWE-bench.

· Updated
· 11 min read
coding-agents agent-evaluation swe-bench claude-code cursor cline aider 2026
Banner with the title Evaluating Coding Agents 2026: A Five-Dimension Eval beside a line diagram linking a coding agent to golden-PR replay, tool calls, multi-file, plan, rollback and clean checkout.
Table of Contents

To run a coding agent evaluation, replay your own merged pull requests against the agent and score every run on five dimensions: golden-PR replay, tool-call correctness, multi-file coherence, plan coherence, and rollback discipline. A public leaderboard cannot tell you whether Cursor, Claude Code, Cline, Aider, Continue, or Replit will ship clean patches against your repo.

Treat public benchmarks as a screen. OpenAI stopped reporting SWE-bench Verified after finding flawed tests and training contamination. It then backed SWE-bench Pro, and in July 2026 retracted that recommendation after estimating that about 30 percent of its tasks are broken. Future AGI supplies the graders, traces, and gateway the steps below run on.

TL;DR

  • OpenAI stopped reporting SWE-bench Verified and later retracted its SWE-bench Pro recommendation, so treat public scores as a screen.
  • Build evals for coding agents from your last 50 merged PRs and replay each one against every candidate agent.
  • Run each task several times from a clean checkout and report pass^k.

Why SWE-bench scores don’t transfer to your codebase

Public benchmarks are designed to be neutral. Your codebase is the opposite. SWE-bench Verified is 500 curated problems from GitHub issues in 12 open-source Python repositories. Your repo has framework conventions the benchmark doesn’t know, internal libraries that aren’t on PyPI, a stricter review bar, and dependencies the agent has never seen.

Four reasons the score doesn’t transfer:

Contamination risk. OpenAI found that every frontier model it tested could reproduce the gold patch or problem details for some SWE-bench tasks. That exposure inflates the score and tells you nothing about novel work. The LLM benchmarks vs production evals post walks through the contamination detection patterns and why you treat any pre-cutoff public score as diagnostic, not procurement-grade.

Surface mismatch. SWE-bench gives the agent a clean repo, a test command, and an issue. Your engineers give the agent a half-loaded VS Code workspace with a partial diff, a Slack thread, and a comment from a reviewer. The agent that wins the bench-rig loses the workflow rig and vice versa.

Idiom drift. Every codebase has conventions a public benchmark can’t model: your custom ORM wrapper, your in-house decorator, your test fixture style, your forbidden imports. Agents that score well on idiomatic-Python tasks can produce diffs your reviewers reject on first read.

Review-bar gap. SWE-bench passes a patch if the failing tests now pass and the existing tests still pass. Your team passes a patch if it makes the failing test pass and doesn’t reformat the file, doesn’t add unrelated imports, and is small enough to review in 90 seconds. Test-based benchmarks score only the tests.

Vendor benchmarks provide signals. The eval that matters runs on your code, with your review bar, on the tickets you actually ship. Pair this post with agent observability vs evaluation vs benchmarking for the full distinction.

Which coding agent benchmarks still matter

Public benchmarks still help you build a shortlist. SWE-bench Pro has 1,865 problems from 41 actively maintained repositories, with long-horizon tasks that often need multi-file patches. OpenAI withdrew its endorsement after estimating that about 30 percent of those tasks are broken. Terminal-Bench grades agents on hard tasks in computer terminal environments and ranks each model together with the agent that ran it, such as Claude Code or Codex.

A score belongs to a model and its harness together, as our breakdown of how the harness changes the score shows. Treat each public number as a screen and the golden-PR replay below as the procurement test.

The five dimensions of a coding agent evaluation

A coding agent evaluation scores five dimensions: golden-PR replay, tool-call correctness, multi-file coherence, plan coherence, and rollback discipline. Each one catches a failure mode that public benchmarks miss, so run all five on every replay instead of trusting a single pass rate.

DimensionWhat it answersGrounded on
Golden-PR replayDoes the agent’s diff pass your review bar against a known-good PR?TaskCompletion + diff comparator
Tool-call correctnessDid search, edit, run-tests, and commit happen in the right order?EvaluateFunctionCalling over the trace
Multi-file coherenceDid the agent update every call site, import, and config that needed it?CustomLLMJudge walking the import graph
Plan coherenceDid the agent’s written plan match the user intent and stay stable across edits?CustomLLMJudge on plan vs final diff
Rollback disciplineDid the agent revert broken changes or pile on top of test failures?Trace-timeline analysis + edits-per-failure ratio

Dimensions one through three gate correctness. Plan coherence gates intent. Rollback discipline gates production safety. The next five sections show how each one composes from primitives the Future AGI evals SDK ships out of the box.

1. Golden-PR replay with your last 50 merges as a test set

The strongest signal you can build against a coding agent is the work you’ve already merged. Your last 50 PRs ship with everything a public benchmark can’t supply: real intent, reviewer feedback, a passing test suite, and a known-good diff. The replay is the contract.

A flow where the last 50 merged pull requests are captured, the repo is reverted to each parent commit for the candidate agent, and the agent's patch is diffed against the merged patch and scored by review-bar match, with the set refreshed weekly.

The protocol:

  1. Pick the last 50 merged PRs, skipping version bumps and generated files. Keep the mix close to what you actually ship.
  2. For each PR, capture the parent commit, the issue text or commit message, the merged diff, and the test deltas.
  3. Revert the repo to the parent. Hand the issue or commit message to the candidate agent. Let it run unobserved against your sandbox.
  4. Diff the agent’s patch against the merged patch. Score by review-bar match, not exact-text match.

The judge is the part most teams get wrong. Exact-match diff scoring punishes agents for plausible variations a human reviewer would have merged. The cleaner pattern is TaskCompletion calibrated against the user intent in the PR description, paired with a coherence judge that compares the agent’s diff to the merged diff for behavioral equivalence.

from fi.evals import evaluate
from fi.evals.metrics import CustomLLMJudge
from fi.evals.llm import LiteLLMProvider

completion = evaluate("task_completion", input=pr.issue_text, output=agent_final_patch, model="turing_flash")

replay_judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={"name": "golden_pr_replay", "grading_criteria": (
        "Score 1.0 if the agent's patch would pass the same code review "
        "as the merged patch. Behavioral equivalence over textual match. "
        "Subtract 0.3 if the agent added unrelated reformats. Subtract "
        "0.5 if the agent's patch fails any test the merged patch passes. "
        "Subtract 0.2 if the diff is more than 2x the size of the merged "
        "diff for the same outcome."
    )},
)

replay = replay_judge.evaluate([{"input": pr.issue_text, "output": agent_final_patch, "context": pr.merged_diff}])

The set ages well. Refresh weekly by promoting one freshly merged PR into the golden set and dropping the oldest. For the broader design pattern, see LLM eval golden set design.

2. Tool-call correctness across read, edit, test, and commit

A coding agent runs a tool-using loop. The agent reads files, edits them, runs tests, reacts to failures, and commits the work. Tool-call correctness asks whether each call happened, in the right order, with the right arguments. Score it from the trace.

Four tool-call behaviors carry the weight:

  • Search before edit. Did the agent open the files it touched before it wrote to them? An agent that edits a file it never read is hallucinating against your code.
  • Edit on target. Did the patch land in the file the plan named? Mis-targeted edits cause cross-file regressions.
  • Run tests after edit. Did the agent run the test runner after the edit, or did it commit blind? An agent that skips the tests is a text editor with extra steps.
  • Commit only what changed. Did the commit boundary match the edit boundary, or did the agent stage unrelated files?

EvaluateFunctionCalling checks that each call was needed and carried the right arguments. Trace the session with traceAI, then score against the call sequence:

calls = evaluate("evaluate_function_calling", input=pr.issue_text, output=tool_calls_json, model="turing_flash")

Future AGI types every span in gen_ai.span.kind. In a traced Claude Code session each prompt is an agent span, each model request an LLM span, and each tool call, such as Read or Bash, a tool span. Claude Code session tracing shows the full span map.

3. Multi-file coherence and the import graph

Single-file diffs hide the failure mode reviewers hate most. The agent renames a helper in module A and leaves seventeen broken call sites in modules B and C. The tests in module A pass and the repo is still wrecked. Multi-file coherence catches it before merge.

The protocol is mechanical. Take the agent’s diff, walk the import graph from every changed file, and produce a list of touched-or-should-have-touched files. For each file in the list, ask a code-review judge whether it’s consistent with the diff.

coherence_judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={"name": "multi_file_coherence", "grading_criteria": (
        "Walk the import graph from every changed file. For each "
        "downstream file, score consistency with the diff. Score 1.0 if "
        "all imports resolve, all symbols referenced exist, no public "
        "API surface is silently removed, and no call site of a renamed "
        "symbol is left stale. Subtract 0.4 per broken import. Subtract "
        "0.6 per stale call site of a renamed public symbol. Subtract "
        "0.3 per config or environment update that should have followed "
        "the code change but didn't."
    )},
)

Multi-file coherence is where Cursor’s Agent mode, Claude Code’s plan mode, Cline’s MCP-driven loop, and Aider’s repo-map approach show their real differences. None of these strategies is correct in the abstract; the eval tells you which one fits your codebase.

4. Plan coherence through the whole loop

Claude Code and Aider can both write a plan before they touch the repo. Claude Code’s plan mode makes no edits until you approve, and Aider’s architect mode has one model propose changes and an editor model turn them into file edits. The plan is content, so score it.

Plan coherence asks two questions: does the plan match the user’s intent at the start, and does the final diff still match the plan at the end? Agents drift mid-loop: sub-task three contradicts the request, the unit tests pass, and the final patch is wrong. A rubric that scores intent against the final diff, and the final diff against the plan, catches drift early.

plan_judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    config={"name": "plan_coherence", "grading_criteria": (
        "Score the agent's plan against the user intent. Then score the "
        "final diff against the plan. Score 1.0 if both alignments are "
        "exact. Subtract 0.3 if the plan dropped a requirement from the "
        "intent. Subtract 0.5 if the final diff added behavior the plan "
        "did not promise. Subtract 0.4 if the plan and diff diverged "
        "after a mid-loop tool failure."
    )},
)

5. Rollback discipline when tests fail

The hardest signal to measure from a chat rubric is whether the agent knows when to undo. Rollback discipline is the behavior at the moment tests fail after an edit. The disciplined agent reverts the broken change and tries a different approach. The undisciplined agent pours more code on top of the failure until the test suite is in pieces.

Score it from the trace timeline. Two numbers do the work: edits per test failure, and the ratio of reverts to retries. Pull both from the span sequence.

def rollback_score(trace):
    failures = [s for s in trace if s.kind == "TOOL"
                and s.name == "run_tests" and s.failed]
    edits_after_failure = []
    for f in failures:
        next_test = next((s for s in trace if s.kind == "TOOL"
                          and s.name == "run_tests"
                          and s.start > f.start), None)
        edits_between = [s for s in trace if s.kind == "TOOL"
                         and s.name in ("edit_file", "revert")
                         and f.start < s.start < (next_test.start
                         if next_test else float("inf"))]
        edits_after_failure.append(edits_between)
    avg_edits = sum(len(e) for e in edits_after_failure) / max(1, len(failures))
    reverts = sum(1 for e in edits_after_failure
                  if any(s.name == "revert" for s in e))
    revert_ratio = reverts / max(1, len(failures))
    return {"avg_edits_per_failure": avg_edits,
            "revert_ratio": revert_ratio}

Read the transcripts of any agent that stacks edits on a failing test without a revert. The AI agent failure modes post catalogs the rollback failure mode at the cluster level, and Error Feed groups those failures into one issue when replays are traced into an Observe project.

Run each golden PR several times in a clean sandbox

One replay per PR can crown the wrong agent. Anthropic’s guide to agent evals tracks pass^k, the chance that all k trials succeed: an agent that solves a PR 75 percent of the time passes three runs in a row only about 42 percent of the time. Start each trial from a clean checkout. In some internal evals, Anthropic saw Claude gain an unfair edge by reading git history from earlier trials.

Production observability with traceAI

A 50-PR replay is offline. Production runs forever. The eval that survives the first month streams from the agent loop in real time. traceAI auto-instruments the model calls your agent code makes, and the gateway reports cost, latency, and routing in response headers.

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from traceai_anthropic import AnthropicInstrumentor
from traceai_openai import OpenAIInstrumentor
from traceai_bedrock import BedrockInstrumentor

trace_provider = register(
    project_type=ProjectType.OBSERVE,
    project_name="coding-agent-eval",
)

AnthropicInstrumentor().instrument(tracer_provider=trace_provider)
OpenAIInstrumentor().instrument(tracer_provider=trace_provider)
BedrockInstrumentor().instrument(tracer_provider=trace_provider)

Once registered, every Anthropic, OpenAI, or Bedrock call your agent code makes becomes an OpenTelemetry span. Claude Code and Cursor run outside your process, so Future AGI ingests Claude Code’s own OpenTelemetry export and Cursor’s hook events instead. To add cost, point Claude Code at the gateway:

export ANTHROPIC_BASE_URL=https://gateway.futureagi.com
export ANTHROPIC_AUTH_TOKEN=sk-agentcc-<per-developer-virtual-key>

The gateway returns headers the eval pipeline reads:

  • x-agentcc-cost: estimated cost in USD, 0 on exact cache hits
  • x-agentcc-latency-ms: end-to-end including provider plus gateway hop
  • x-agentcc-model-used: what routing actually picked
  • x-agentcc-fallback-used: true when a provider fallback occurred
  • x-agentcc-routing-strategy: such as cost-optimized or least-latency

The per-developer virtual key gives attribution without exposing the raw provider key, so “which engineer’s Claude Code sessions cost the most last week” is one query. For the cost math at scale, how to reduce Claude Code token costs 90 percent walks through the lever sizing.

Security regression checks with four scanners on every diff

A coding agent can paste a credential into a config file or reach for shell=True to make a test pass. Four local scanners in agent-learning-kit each check a patch in under 10 ms with no API key:

from fi.evals.guardrails.scanners import (
    ScannerPipeline, SecretsScanner, CodeInjectionScanner,
    JailbreakScanner, RegexScanner, RegexPattern,
)

pipeline = ScannerPipeline([
    SecretsScanner(),
    CodeInjectionScanner(),
    JailbreakScanner(),
    RegexScanner(custom_patterns=[
        RegexPattern(name="python_eval", pattern=r"\beval\("),
        RegexPattern(name="shell_true", pattern=r"subprocess\.\w+\(.*shell=True"),
    ]),
])

scan_result = pipeline.scan(agent_final_patch)

SecretsScanner catches credentials that land in the patch. CodeInjectionScanner flags SQL, shell, path traversal, and template injection. JailbreakScanner runs against comments and docstrings, since multi-turn sessions sometimes round-trip user content into generated text. RegexScanner flags the repo’s banned patterns, and the OWASP LLM Top 10 post covers the risks behind these checks.

Where Future AGI fits

Future AGI is the eval, observability, and gateway stack for teams that build, operate, or pick coding agents. Three pieces do the work:

  • traceAI: an Apache-2.0 OpenTelemetry SDK that instruments AI apps in Python, TypeScript, Java, and Go, with manual spans in C#.

  • Evals: built-in templates, LLM-as-Judge, and local scanners in the Apache-2.0 agent-learning-kit SDK (pip install agent-learning-kit).

  • Agent Command Center: an OpenAI-compatible LLM gateway in the Apache-2.0 core, shipped as a Go binary and Docker image, with 100+ providers and 18 built-in guardrail scanners plus 15 vendor adapters.

Future AGI is SOC 2 Type II and ISO 27001 certified per futureagi.com/trust, and HIPAA compliant with a BAA available. Set a threshold on any eval score and it becomes a merge gate in CI/CD, so an agent upgrade that regresses on your golden PRs is blocked before it ships.

The agent vendor owns the prompt. You own the trace, the eval, the cost, and the routing. To pick the metric behind each dimension, see agent metrics frameworks 2026.

Choosing a coding agent on your own code

A top public score tells you an agent can fix issues in someone else’s repo. Your last 50 merged PRs tell you whether it can fix yours. Replay them, score the five dimensions in a clean checkout, and route model calls through the gateway so quality and cost land on the same eval row.

Start on the Future AGI Free plan, no credit card needed, and run your first golden-PR replay this week.

Sources

Frequently Asked Questions

How do you evaluate AI coding agents?

Evaluate AI coding agents by replaying your own merged pull requests against each agent, then grading the patch, the tool-call sequence, and the plan it wrote. Run unit tests for correctness, add an LLM judge for review quality, and repeat each task in a clean sandbox.

Is SWE-bench Verified still a good coding agent benchmark?

SWE-bench Verified is now a weak signal, because OpenAI stopped reporting it after finding flawed tests and evidence that frontier models had seen its problems in training. OpenAI later retracted its SWE-bench Pro recommendation after estimating about 30 percent of its tasks are broken. Use either as a screen, then run a coding agent evaluation on your own repo.

What should a coding agent eval measure?

A coding agent eval should measure whether the patch passes your review bar and whether the agent searched, edited, tested, and committed in the right order. Add multi-file coherence for call sites the diff missed, plan coherence for drift from the request, and rollback discipline for how the agent reacts when tests fail.

Should you grade coding agents with unit tests or an LLM judge?

Use both: unit tests grade correctness because a patch either passes or fails, and an LLM judge grades what tests miss, like scope, style, and tool use. Anthropic's agent eval guide describes the same split. Calibrate the judge against human reviewers before trusting it.

What is a coding agent evaluation framework?

A coding agent evaluation framework is a repeatable test plan: a task set from your repo, a grader per failure mode, and a fixed number of runs per task. Start with your last 50 merged PRs, skipping version bumps and generated files. Future AGI runs the graders and traces each run, so scores and spans sit together.
Related Articles
View all