What Is an Eval Harness and Why Your Score Depends on It

Every benchmark score you read was produced by software that made many small choices along the way. This is the plumbing behind that number, and why a score is only as trustworthy as the harness that produced it.

Two teams evaluate the same open model on MMLU, the 57-subject multiple-choice knowledge test. One team reports 63.7% accuracy; the other reports 48.8%. Nobody made a mistake, and both ran the same benchmark on the same model weights. The gap of nearly 15 points came entirely from the eval harness each team used.

An eval harness is the software that turns a benchmark into a score. It loads the dataset, builds the prompts, decides how many few-shot examples to show, calls the model, reads the output, computes a metric, and writes the result to a log. Swap any of those steps and the number moves, which is why the harness deserves as much attention as the model it measures.

This guide walks through how an eval harness works from end to end. You will see the three stages a harness runs, the settings that decide the score, the output types that define a correct answer, how it differs from an agent harness and from guardrails, and how academic harnesses differ from production ones.

TL;DR

• An eval harness runs a test set through a model or app under fixed settings, scores every output, and logs the run so another team can rerun the same protocol.
• The same model can score about 15 points apart on MMLU depending only on how the harness asks the question and scores the answer.
• Academic harnesses rank models on public benchmarks; production harnesses run your own cases as a release gate.
• An eval harness measures. An agent harness runs the agent. A guardrail intervenes in a live request.

What Is an Eval Harness?

An eval harness is the software layer that turns a benchmark into a reproducible score. It loads the dataset, formats the prompts with any few-shot examples, calls the model, parses the raw output, computes a metric, and logs the result.

The word reproducible is the whole point. If you hand the same benchmark, the same model, and the same harness settings to another team, they can rerun the exact protocol, and with deterministic settings (greedy decoding or likelihood scoring) they should get the same number. The harness is what makes a score repeatable instead of a one-off demo, and it is what makes two scores comparable, but only when both were produced the same way.

How an Eval Harness Differs From an Agent Harness

An agent harness runs an agent so it can act. An eval harness measures a model or an agent against a fixed test.

Both wrap a model in software, which is why the two get blurred. An agent harness wraps a model in a loop, gives it tools, and manages memory so it can finish a task; this guide covers what an agent harness is and what it controls. An eval harness has a narrower job: present a fixed set of questions or tasks the same way every time and score the answers. One runs a system; the other grades it.

When the thing under test is an agent, the two meet. The agent harness, with its prompts, tools, memory, retries and loop control, becomes part of the system being measured, and the eval harness sits outside it, scoring the trajectory, the tool calls and the final outcome. Change the agent harness and the score can move even though the model did not change. For evaluating agents, where tool calls and trajectories come into play, see the agent eval harness.

How Does an Eval Harness Work?

An eval harness runs a three-stage pipeline: the inputs (what gets evaluated), the execution (how it gets scored), and the actions (what happens when the scores come back). The same arc applies to a research benchmark runner and to a production test suite. The three stages are where all the important choices live.

Inputs: Tasks, Prompts, and Few-Shot Setup

The input stage decides what the model actually sees. It loads the task dataset, wraps each example in a prompt template, and prepends any few-shot examples that show the model the expected format. A task like HellaSwag arrives as raw sentence-completion pairs, and the harness turns each one into a formatted prompt with a fixed number of worked examples in front of it. Change the template or the shot count and you have changed the test.

Execution: Calling the Model and Decoding

Execution is the model call itself, plus the decoding settings that turn logits into text. The harness sends each prompt to the model and collects the response under fixed generation settings, such as greedy decoding or a set sampling temperature. Holding these constant is what keeps a run repeatable, since the same prompt under sampled decoding can produce different answers on different runs.

Actions: Metrics, Aggregation, and Logging

The action stage converts raw outputs into a score and records it. The harness parses each answer, applies a metric like exact match or accuracy, aggregates across every example, and writes the results to a file. Logging matters here, because a score with no record of the prompts, the shot count, and the metric behind it cannot be reproduced or trusted later.

How Eval Harness Settings Change LLM Benchmark Scores

The harness fixes five things that decide the number: the prompt template, the few-shot count, the output type, the metric, and how results are aggregated. Change any one and the same model on the same benchmark returns a different score. This is why a bare benchmark name tells you little on its own.

The clearest evidence comes from MMLU. When Hugging Face traced why one model scored so differently across libraries, they found that a single LLaMA 65B model scored 0.488 on EleutherAI’s harness, 0.637 on HELM, and 0.636 on the original implementation, a spread of roughly 15 points for identical weights (Hugging Face documented the full breakdown in June 2023, on the harness versions of that time).

The gap came from how each harness asked the question. The original implementation compared the probabilities of just the answer letters A, B, C, and D. HELM had the model generate the letter as text and checked it, while EleutherAI’s harness compared the probability of the full answer sentence, letter and text together. Same model, same questions, three different definitions of a correct answer, three different scores.

The few-shot count is another knob that moves the number. A zero-shot run, with no worked examples, can score differently from a five-shot run, where the model sees the format a few times first. That is why a reported number should always say how many shots produced it, since a zero-shot and a five-shot result describe two different tests of the same model.

The harness maintainers make the same point from the other side: the lm-evaluation-harness README says publicly available prompts are what make results reproducible and comparable between papers. Change the prompt and you are running a different test, so document the changed prompt with the score. Harness details belong next to every number you cite, because the number alone cannot tell you how it was produced.

Is an Eval Harness Deterministic?

Not always, and the difference between reproducible and deterministic matters. A reproducible run is one whose protocol another team can recover and rerun: the same dataset, prompts, settings and scoring. A deterministic run goes further and returns identical outputs every time.

Greedy decoding and likelihood scoring (loglikelihood, multiple_choice) give repeatable scores. Sampled generation, tool-using agents and LLM judges can vary from run to run, depending on how they are configured, so run repeated trials and report the spread instead of a single number.

The harness can also shift measured capability. OpenAI’s third-party evaluation playbook notes that the harness can change how a system uses tools, keeps track of information, or recovers from mistakes, and reports that GPT-5.5 performed better on OpenAI’s cyber ranges when the harness used compaction to preserve task-relevant context.

Record these so two runs are comparable:

• Dataset version and split
• Prompt template and shot count
• Model snapshot
• Decoding settings
• Output type and answer parser
• Metric and pass threshold
• Grader model and rubric, if an LLM judge scores the output
• Number of trials
• Any excluded cases

Output Types: How the Harness Scores an Answer

An output type is the harness’s rule for what counts as an answer and how to score it. EleutherAI’s lm-evaluation-harness defines four, and the choice decides how a correct answer is even measured. The names below are the exact terms used in the tool.

EleutherAI’s lm-evaluation-harness uses generate_until for tasks where the model writes free-form text that a metric then parses, such as a math word problem with a final numeric answer. loglikelihood scores by comparing the probability the model assigns to a target string against alternatives, which suits tasks with one known answer. loglikelihood_rolling sums the log-probabilities across a whole sequence and drives perplexity-style tasks. multiple_choice ranks the answer options by likelihood and picks the top one.

Probability-based scoring has a practical draw. Because loglikelihood and multiple_choice read the probabilities the model assigns instead of asking it to generate free text, they run fast and give repeatable scores, which makes large multiple-choice benchmarks like MMLU cheap to run and easy to reproduce. Open-ended generate_until tasks cost more, because the harness has to generate and then parse real text.

The reason this matters is the definition of correct. A model that looks strong under multiple_choice, where it only has to rank four options, may score lower under generate_until, where it has to produce the exact answer text unaided. When you read a benchmark result, the output type tells you how hard the scoring was.

Are Online Evals the Same as Guardrails?

Evals and guardrails answer different questions at different moments. An eval harness measures quality against a fixed dataset to tell you whether a model or app is good enough to ship. Guardrails run online in the live request path, checking or blocking each input and output in real time. One is measurement; the other is enforcement.

Timing is one way to tell them apart. Evals run before and after deployment, on a dataset you control, with no user waiting on the result. A guardrail runs inside the production request, in the moment, and has to decide fast enough that a real user does not notice the check. That is why a guardrail leans on quick classifiers and rules, while an eval can afford a slower, more thorough metric.

A toxicity check makes the split concrete. A toxicity eval might score a thousand saved prompts overnight and report a pass rate you read the next morning. A toxicity guardrail inspects the single response a user is about to see and blocks it in the moment if it crosses a threshold. Same concern, two very different jobs.

The line is real but not absolute. Online evals exist too, where you score a sample of live production traffic to watch quality drift over time. Online versus offline describes when the data arrives; eval versus guardrail describes whether the check can change the response.

The reliable distinction is enforcement: a guardrail is the layer that can block or rewrite an output as it happens, while an eval reports a number that a human or a CI gate then acts on. Keep the two straight and you avoid asking an offline benchmark to do a job only a runtime guardrail can do.

Academic vs Production Eval Harnesses

Academic and production harnesses share the same three stages but serve different goals. Academic harnesses run fixed public benchmarks to compare models on a leaderboard. Production harnesses run your own app-specific test cases, for example as a CI gate that blocks a release when scores drop. The mechanics match, but the datasets and the stakes differ.

On the academic side, EleutherAI’s lm-evaluation-harness and Stanford’s HELM standardize how models get prompted, scored, and logged so results hold up across papers. HELM looks past a single number. It reports seven metrics at once, accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, so a model that leads on accuracy but scores poorly on calibration has nowhere to hide.

On the production side, the goal shifts from ranking models to guarding an app. Production harnesses run app-specific goldens inside a CI suite, so a failing metric can block a release the same way a failing unit test does.

On Future AGI, an eval is a versioned template you write once and reuse by name across a dataset, a simulation, a live trace, or a CI check, so the same check that gated the release keeps scoring production traffic. Academic harnesses answer which model is better in general; production harnesses answer whether this build of your app is safe to ship.

Which one you reach for depends on the question in front of you. Reach for an academic harness when you need to compare models on neutral, public ground, such as picking a base model to build on.

Reach for a production harness when you need to know whether your own app still behaves after a prompt change, a model swap, or a new release. Running both covers the two jobs: one picks the model and one keeps the app honest over time.

Eval Harness Examples

Four harnesses show the range on the academic and research side:

• EleutherAI lm-evaluation-harness: runs over 60 standard academic benchmarks, with hundreds of subtasks and variants.
• Stanford HELM: scores language models on seven metrics (accuracy, calibration, robustness, fairness, bias, toxicity and efficiency) and publishes leaderboards.
• Inspect: a framework for frontier AI evaluations from the UK AI Security Institute and Meridian Labs, with support for agent evaluations, tool calling and sandboxed execution.
• OpenAI Evals: a framework for evaluating LLMs and LLM systems, with an open-source registry of benchmarks and support for writing your own evals.

On the production side, the harness is your own test set plus the evals that score it. A support chatbot team might keep a set of real customer questions with approved answers, score every new build for groundedness and instruction adherence, and block the release when the pass rate drops.

Building a Production Eval Harness With Future AGI

Future AGI has custom evals: write your own grading rule, choose an LLM judge or a deterministic check, map dataset columns to its inputs, set a pass/fail threshold, and run it as a CI gate. Built-in evals such as Groundedness, Detect Hallucination, Context Adherence, Task Completion and Instruction Adherence cover grounding, hallucination and task checks without writing criteria from scratch.

1. Pick a built-in eval, or write a custom one with your own criteria.
2. Choose the output type: pass or fail, a score, or a category.
3. Set the pass threshold that turns a score or category into a pass or fail.
4. Test it on sample data in the eval’s Test Evaluation panel, and read the result and its explanation before you attach it anywhere.
5. Save it as a template. Each edit is kept as a numbered version, so you can pin a run to one version and it scores the same way when you rerun it later.
6. Run it where your data lives: across a dataset, on live traces, in a simulation, or as a check on every pull request.

Future AGI Evaluate gives the product view, and the platform’s Apache-2.0 core is self-hostable from the future-agi repository. For local, code-first runs, Agent Learning Kit, Future AGI’s Apache-2.0 SDK and CLI (pip install agent-learning-kit), runs the same kind of evals from your own code.

Reading a Score Through Its Harness

Go back to the two teams from the start, one reporting 63.7% on MMLU and the other 48.8% for the same model. With the harness in view, the gap stops being a mystery. One harness scored the full answer sentence, the other scored answer letters, and that single choice moved the number nearly 15 points. The model never changed.

That is the whole idea of an eval harness. It runs the three stages, inputs, execution, and actions, and along the way it fixes the prompt template, the few-shot count, the output type, and the metric. Each of those choices shapes the score, which is why the same model can land in very different places depending on the harness that measured it.

So when you meet a benchmark number, read the harness first. Ask which output type produced it, how many few-shot examples were used, whether the run was academic or production, and where the result was logged. A score with those details attached is something you can trust and compare. A score without them is just a number.

The practical guide to lm-evaluation-harness walks through a first evaluation, and adding a custom task shows how to test a model on your own data. If you are building that harness for your own app, Future AGI lets you save each eval as a versioned template and run the same check on datasets, traces and in CI.