What Is an Eval Harness? LLM Evaluation, Explained

Every benchmark score you read was produced by software that made dozens of small choices along the way. This is the plumbing behind that number, and why a score is only as trustworthy as the harness that produced it.

Two teams evaluate the same open model on MMLU, the 57-subject multiple-choice knowledge test. One team reports 63.7% accuracy. The other reports 48.8%. Nobody made a mistake, and both ran the same benchmark on the same model weights. The gap of nearly 15 points came entirely from the eval harness each team used.

An eval harness is the software that turns a benchmark into a score. It loads the dataset, builds the prompts, decides how many few-shot examples to show, calls the model, reads the output, computes a metric, and writes the result to a log. Swap any of those steps and the number moves, which is why the harness deserves as much attention as the model it measures.

This post explains what an eval harness is and how one works from end to end. It covers the three stages every harness runs, the settings that decide the score, the output types that define a correct answer, how evals differ from guardrails, and how academic harnesses differ from production ones.

The Direct Answer: An Eval Harness Runs the Benchmark

An eval harness is the software layer that turns a benchmark into a reproducible score. It loads the dataset, formats the prompts with any few-shot examples, calls the model, parses the raw output, computes a metric, and logs the result. An agent harness runs an agent so it can act. An eval harness measures a model against a fixed test.

That distinction matters, because the two often get blurred. An agent harness wraps a model in a loop, gives it tools, and manages memory so it can finish a task. An eval harness has a narrower job: present a fixed set of questions the same way every time and score the answers. One runs a system; the other grades it.

The word reproducible is the whole point. If you hand the same benchmark, the same model, and the same harness settings to another team, they should get the same number. The harness is what makes a score repeatable instead of a one-off demo, and it is what makes two scores comparable, but only when both were produced the same way. For evaluating agents, where tool calls and trajectories come into play, see the agent eval harness.

The Three Stages Every Eval Harness Runs

Arize describes an eval harness as a three-stage pipeline: the inputs (what gets evaluated), the execution (how it gets scored), and the actions (what happens when the scores come back). Every harness follows this arc, from a research benchmark runner to a production test suite. The three stages are where all the important choices live.

Inputs: Tasks, Prompts, and Few-Shot Setup

The input stage decides what the model actually sees. It loads the task dataset, wraps each example in a prompt template, and prepends any few-shot examples that show the model the expected format. A task like HellaSwag arrives as raw sentence-completion pairs, and the harness turns each one into a formatted prompt with a fixed number of worked examples in front of it. Change the template or the shot count and you have changed the test.

Execution: Calling the Model and Decoding

Execution is the model call itself, plus the decoding settings that turn logits into text. The harness sends each prompt to the model and collects the response under fixed generation settings, such as greedy decoding or a set sampling temperature. Holding these constant is what keeps a run repeatable, since the same prompt under sampled decoding can produce different answers on different runs.

Actions: Metrics, Aggregation, and Logging

The action stage converts raw outputs into a score and records it. The harness parses each answer, applies a metric like exact match or accuracy, aggregates across every example, and writes the results to a file. Logging matters most here, because a score with no record of the prompts, the shot count, and the metric behind it cannot be reproduced or trusted later.

What the Harness Controls, and Why the Score Depends on It

The harness fixes five things that decide the number: the prompt template, the few-shot count, the output type, the metric, and how results are aggregated. Change any one and the same model on the same benchmark returns a different score. This is why a bare benchmark name tells you almost nothing on its own.

The clearest evidence comes from MMLU. When Hugging Face traced why one model scored so differently across libraries, they found that a single LLaMA 65B model scored 0.488 on EleutherAI’s harness, 0.637 on HELM, and 0.636 on the original implementation, a spread of roughly 15 points for identical weights (Hugging Face documented the full breakdown).

The gap came from how each harness asked the question. The original implementation compared the probabilities of just the answer letters A, B, C, and D. HELM had the model generate the letter as text and checked it. EleutherAI’s harness compared the probability of the full answer sentence, letter and text together. Same model, same questions, three different definitions of a correct answer, three different scores.

The few-shot count is a smaller knob that moves the number the same way. Running a benchmark zero-shot, with no worked examples, usually scores lower than running it five-shot, where the model sees the format a few times first. That is why a reported number should always say how many shots produced it, since a zero-shot and a five-shot result describe two different tests of the same model.

The lesson is one the harness maintainers state plainly: a score is tied to its implementation, down to the prompt and the tokenization. EleutherAI even discourages comparing runs across different papers for this reason. So harness details belong next to every number you cite, because the number alone cannot tell you how it was produced.

Output Types: How the Harness Scores an Answer

An output type is the harness’s rule for what counts as an answer and how to score it. EleutherAI’s lm-evaluation-harness defines four, and the choice decides how a correct answer is even measured. The names below are the exact terms used in the tool.

The lm-evaluation-harness uses generate_until for tasks where the model writes free-form text that a metric then parses, such as a math word problem with a final numeric answer. loglikelihood scores by comparing the probability the model assigns to a target string against alternatives, which suits tasks with one known answer. loglikelihood_rolling sums the log-probabilities across a whole sequence and drives perplexity-style tasks. multiple_choice ranks the answer options by likelihood and picks the top one.

Probability-based scoring has a practical draw. Because loglikelihood and multiple_choice read the probabilities the model assigns instead of asking it to generate free text, they run fast and stay fully deterministic, which makes large multiple-choice benchmarks like MMLU cheap to run and easy to reproduce. Open-ended generate_until tasks cost more, because the harness has to generate and then parse real text.

The reason this matters is the definition of correct. A model that looks strong under multiple_choice, where it only has to rank four options, may score lower under generate_until, where it has to produce the exact answer text unaided. When you read a benchmark result, the output type tells you how hard the scoring actually was.

Eval Harness vs Guardrails: Offline and Online

Evals and guardrails answer different questions at different moments. An eval harness measures quality against a fixed dataset, usually offline, to tell you whether a model or app is good enough to ship. Guardrails run online in the live request path, checking or blocking each input and output in real time. One is measurement; the other is enforcement.

The timing is the cleanest way to tell them apart. Evals run before and after deployment, on a dataset you control, with no user waiting on the result. A guardrail runs inside the production request, in the moment, and has to decide fast enough that a real user does not notice the check. That is why a guardrail leans on quick classifiers and rules, while an eval can afford a slower, more thorough metric.

A toxicity check makes the split concrete. A toxicity eval might score a thousand saved prompts overnight and report a pass rate you read the next morning. A toxicity guardrail inspects the single response a user is about to see and blocks it in the moment if it crosses a threshold. Same concern, two very different jobs.

The line is real but not absolute. Online evals exist too, where you score a sample of live production traffic to watch quality drift over time. The reliable distinction is enforcement: a guardrail is the layer that can actually block or rewrite an output as it happens, while an eval reports a number that a human or a CI gate then acts on. Keep the two straight and you avoid asking an offline benchmark to do a job only a runtime guardrail can do.

Academic vs Production Eval Harnesses

Academic and production harnesses share the same three stages but serve different goals. Academic harnesses run fixed public benchmarks to compare models on a leaderboard. Production harnesses run your own app-specific test cases, often as a CI gate that blocks a release when scores drop. The mechanics match, but the datasets and the stakes differ.

On the academic side, EleutherAI’s lm-evaluation-harness and Stanford’s HELM standardize how models get prompted, scored, and logged so results hold up across papers. HELM makes the point that a single number is rarely enough. It reports seven metrics at once, accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency, so a model that leads on accuracy but scores poorly on calibration has nowhere to hide.

On the production side, the goal shifts from ranking models to guarding an app. Harnesses like DeepEval run app-specific goldens inside a pytest-style suite, so a failing metric can block a release the same way a failing unit test does. Platforms such as Braintrust pair evals with observability to track the same checks in production. Academic harnesses answer which model is better in general; production harnesses answer whether this build of your app is safe to ship.

Which one you reach for depends on the question in front of you. Reach for an academic harness when you need to compare models on neutral, public ground, such as picking a base model to build on. Reach for a production harness when you need to know whether your own app still behaves after a prompt change, a model swap, or a new release. Most teams end up running both, one to choose the model and one to keep the app honest over time.

Reading a Score Through Its Harness

Go back to the two teams from the start, one reporting 63.7% on MMLU and the other 48.8% for the same model. With the harness in view, the gap stops being a mystery. One harness scored the full answer sentence, the other scored answer letters, and that single choice moved the number nearly 15 points. The model never changed.

That is the whole idea of an eval harness. It runs the three stages, inputs, execution, and actions, and along the way it fixes the prompt template, the few-shot count, the output type, and the metric. Each of those choices shapes the score, which is why the same model can land in very different places depending on the harness that measured it.

So when you meet a benchmark number, read the harness first. Ask which output type produced it, how many few-shot examples were used, whether the run was academic or production, and where the result was logged. A score with those details attached is something you can trust and compare. A score without them is just a number. The practical guide to lm-evaluation-harness walks through a first evaluation, and adding a custom task shows how to test a model on your own data.