Coding Agent Harness Benchmarks: Why the Harness Changes the Score

A leaderboard number is a joint result of the model and the harness around it. Read it that way, and two contradictory “same model” scores stop being a contradiction.

You are comparing two coding agents. One team’s harness reports a SWE-bench Pro pass rate in the low 50s. An independent, standardized run of the very same model lands near 45. Nobody faked anything, and both numbers are real measurements. The gap is the harness, and once you see it you cannot unsee it.

A coding agent harness is the scaffolding that turns a raw model into something that can act on a repository: the prompts, the tool set, the retry logic, the loop control, and the context strategy. Swap that scaffolding and the same model shows a different amount of its ability. So a benchmark score always carries two signals at once: the model, and the harness that ran it.

This post is about reading those numbers honestly. It covers why the same model scores differently across coding agent harness benchmarks, where the evidence comes from, which leaderboards try to isolate the harness effect, and a short checklist for judging any coding-agent score you meet. Every figure here is timestamped and sourced, because these numbers move fast.

The Direct Answer: A Score Is a Model-Plus-Harness Result

A benchmark pass rate reflects two things at once, the model and the scaffold that runs it. The scaffold means the prompts, the tool set, the retry policy, the loop control, and the context strategy. Hold the model fixed and the number still moves as the harness changes. So read every score as a model-plus-harness result.

The research term for this is the binding constraint. A 2026 paper, Stop Comparing LLM Agents Without Disclosing the Harness, advances what its authors call the Binding Constraint Thesis: the execution harness is often a stronger determinant of agent performance than the model it wraps. Their claim is blunt. Current evaluation protocols keep crediting harness-level gains to the model, and that misattribution shows up as model-ranking reversals once you change the scaffold.

How big is the design space that produces these swings? A source-code taxonomy of open-source coding agents (Inside the Scaffold) found tool counts ranging from 0 to 37, seven different context-compaction strategies, and five distinct loop primitives across 13 systems. Two harnesses running the identical model can differ that much in how they prompt it, feed it context, and let it retry. That variation is exactly what a single pass rate hides. For the architectural reasons the scaffold carries so much of the outcome, see why most of an agent’s performance lives outside the model.

The practical consequence is simple. When a score changes, the number alone cannot tell you whether the model got better or the harness did. Separating those two is the entire job of reading a benchmark well.

The Evidence: Same Model, Different Harness, Different Score

Hold the model constant and the harness still moves the score. On SWE-bench Pro with Claude Opus 4.5, an independent standardized run reports about 45.9%, while a vendor comparison of three harnesses running the same model reports roughly 49.8% to 51.8%. Same weights, four setups, a spread of several points.

Figures as of publish date, SWE-bench Pro public set, pass@1, Claude Opus 4.5 held constant. The standardized number is Scale AI’s SEAL-lab run; the three vendor-comparison figures come from a single published table by Augment Code. Verify against the live leaderboards before citing.

The lower number is Scale AI’s SEAL-lab standardized evaluation of SWE-bench Pro, which runs models through a common SWE-agent scaffold so the harness stays fixed. The three higher numbers come from a published comparison by Augment Code, which ran the same Opus 4.5 model through several agent harnesses. Two details are worth noticing. The independent standardized run sits below all three vendor-comparison numbers, and the lab that published the table shows its own agent, Auggie, at the top. Neither detail implies foul play. Both are why the source of a score matters as much as the score.

This is not one dataset’s quirk. Reviewing SWE-bench Verified, Epoch AI found that a good scaffold can lift performance by up to 20%, and documented one model’s score ranging from 62.3% to 70.2% purely by scaffold choice. A separate longitudinal study, Don’t Blame the Large Language Model, tracked 35 sequential scaffolding releases with the model held constant and traced the quality swings to the scaffolding itself. Teams kept blaming the model for regressions the harness had caused.

Why Vendor and Independent Numbers Diverge

Three forces open the gap. First, harness differences: one setup gives the model more tools, better retries, or a cleaner context window than another. Second, test-time compute: a vendor run may allow more attempts, longer turn limits, or larger sampling budgets than a standardized run. Third, contamination controls: independent evaluations on a contamination-resistant set like SWE-bench Pro tend to report lower, more conservative numbers than older, more-exposed benchmarks.

None of these require anyone to cheat. They are ordinary configuration choices, and they explain why the same model lands in different places. Secondary write-ups push the claim further, with some reporting swings above 20 points between scaffolds, so treat those as directional and confirm the model and variant were held fixed.

The Leaderboards That Isolate the Harness Effect

A few efforts fight the attribution problem by holding variables constant on purpose. Princeton HAL runs models through one standardized, cost-controlled evaluation harness across nine benchmarks. The Artificial Analysis Coding Agent Index reports a single composite built from disclosed components. Both are more honest than a one-off vendor chart, because the setup is shared and stated out loud.

Princeton HAL, the Holistic Agent Leaderboard, provides a unified evaluation command that runs agents through the same cost-controlled setup across nine benchmarks, among them SWE-bench Verified, USACO, AppWorld, tau-bench, and SciCode. The repository was archived in mid-2026, so treat it as a reference point, but the idea is the durable part. When the evaluation harness is shared and the cost is controlled, differences between agents get easier to attribute to the agent instead of the scaffold around it. HAL was published at ICLR 2026.

The Artificial Analysis Coding Agent Index takes a different route to the same goal. Instead of one benchmark, it publishes a composite score with a disclosed recipe, so you can see exactly what went into the average. The next section is about why that recipe matters more than the headline it produces.

What a Composite Index Actually Averages

A single coding-agent number is often an average of several sub-tests, and the average can hide wide disagreement between them. The Artificial Analysis Coding Agent Index, for example, is a simple mean of pass@1 across three components. Read the components before you trust the headline, because a strong average can sit on top of one weak sub-score.

As of its June 2026 version, the index averages pass@1 across three sub-benchmarks: DeepSWE, with 113 implementation and bug-fix tasks; Terminal-Bench v2, with 84 command-line workflow tasks; and SWE-Atlas-QnA, with 124 repository-understanding questions. That is 321 tasks folded into one figure. A harness that patches code well but stumbles in the terminal can post the same composite as a balanced one, and the headline will not tell you which you are looking at.

This is the general trap with any blended score. Averages compress dispersion. When a leaderboard gives you a composite, find the sub-scores, because the shape of the distribution is where the useful signal lives. If a tool publishes only the composite, treat it as a starting point instead of a verdict.

The SWE-bench Variants You’re Actually Comparing

SWE-bench ships in several sizes under one name, and a score only means something next to another score from the same variant. Comparing a Verified number to a Pro number is an apples-to-oranges mistake that shows up constantly in tool marketing. The table below lays out what each variant actually measures.

The full SWE-bench holds about 2,294 real GitHub issues across 12 Python repositories, each asking the agent to produce a patch that makes hidden tests pass. SWE-bench Verified is a 500-task subset that human annotators, working with OpenAI’s Preparedness team, confirmed to be fair and solvable. SWE-bench Lite trims that to 300 lighter tasks for quick iteration. SWE-bench Multimodal moves to JavaScript and TypeScript issues that need visual context, with roughly 510 test instances.

SWE-bench Pro is the newer, harder set, 731 public tasks out of 1,865 total, built by Scale AI to resist contamination by drawing on copyleft and private commercial repositories (arXiv 2509.16941). Because it is harder and cleaner, scores on Pro run well below scores on Verified for the same system.

So when one page cites a Verified pass rate and another cites a Pro pass rate, the higher number often just reflects the easier test, and the comparison says little about which agent is stronger. If you plan to run these tasks yourself, the SWE-bench evaluation harness has its own setup and Docker gotchas.

How to Read a Coding-Agent Leaderboard Honestly

Before you trust any coding-agent score, run five quick checks. Same benchmark variant? Standardized or vendor harness? Independent source? Timestamped? Single-shot or multi-attempt scoring? If a comparison fails any of these, it is not measuring what it claims to measure. The checklist below turns that into a habit.

Walk the five questions in order. Same variant asks whether both numbers come from, say, Verified and Verified, not Verified and Pro. Standardized or vendor asks who built the harness that produced the score, an independent common scaffold or the tool’s own setup. Independent source asks whether the number comes from a neutral evaluator or the seller. Timestamped asks for a date, because a six-month-old score describes a model and a harness that have both moved since. Single-shot or multi-attempt asks whether the score is pass@1 or pass@k, since more attempts inflate the number.

Any single failed check is enough to void a comparison. Two scores from different variants measure different things. A vendor number without an independent cross-check reads as an upper bound, a ceiling to confirm elsewhere. An undated score is a snapshot of a moment that has already passed. The checklist is quick, and it filters out most of the misleading comparisons you will run into.

Reading the Harness Behind the Number

Go back to the two agents from the start, one harness reporting the low 50s, an independent run of the same model near 45. With the checklist, the contradiction dissolves. Same model, different scaffolds, different sources, and once you know that, you know which number to trust for your own decision.

The through-line of every section here is one idea. A coding agent harness benchmark measures a system, the model and the scaffold together, and the scaffold often moves the number more than a model upgrade would. The binding-constraint research argues it, the same-model comparisons show it, and the leaderboards that hold the harness fixed exist precisely because of it.

So when you need to choose, invert the usual order. Read the harness first, then the model. Confirm the benchmark variant, find an independent standardized number to anchor the vendor claims, check the date, and note the attempt budget. Do that, and a leaderboard stops being a scoreboard you take on faith and becomes something you can reason about. The same discipline carries into the wider machinery of evaluating agents as whole systems: name the setup, hold it fixed, and report what you actually ran.