HumanEval Benchmark Explained: What Pass@k Does and Does Not Prove

A high HumanEval score answers one narrow question well. This post is about the questions it never asked.

A model scores 96% pass@1 on HumanEval. That number gets put on a slide, quoted in a launch post, and used to pick a coding model for a production team. It sounds decisive. It is not.

HumanEval measures one specific thing: can a model write a short, standalone Python function that passes a handful of unit tests. That is a real capability. It is also a long way from writing code inside your repository, with your conventions, against your test suite.

This post covers what the HumanEval benchmark actually is, how the pass@k estimator works, and the three structural limits that a high score hides. It then covers where the benchmark still earns its keep in 2026, and what to run next to it.

The HumanEval Benchmark, Defined

HumanEval is a code generation benchmark of 164 hand-written Python problems. OpenAI released it alongside the Codex paper in 2021 to measure functional correctness rather than surface-level similarity to a reference solution. It has been the default first number in code-model release notes ever since.

Origin: OpenAI, Codex, and 164 Hand-Written Problems

The Codex paper introduced HumanEval as “164 original programming problems with unit tests,” averaging 7.7 tests per problem (arXiv:2107.03374). The problems were written by hand specifically so they would not already sit in a training corpus scraped from GitHub. That was the whole design intent, and it is the part that aged worst.

For scale, Codex-12B solved 28.8% of the problems on a single sample in that original paper. Allowed 100 samples per problem, the same system reached 70.2%. Those two numbers on one model, five years ago, already show how much the reported figure depends on how many attempts you count.

How a Problem Is Structured and Scored

Each HumanEval item ships as five fields: a task_id, a prompt containing the function signature and docstring, a canonical_solution, a test function, and an entry_point naming the function under test (dataset card). The model sees only the prompt and writes the body.

Scoring is execution-based. The generated function is run against the hidden tests, and it either passes all of them or it fails. There is no partial credit, no diff against the canonical solution, and no style judgment. That binary pass/fail is what makes the benchmark cheap and reproducible, and it is also what makes it blunt.

What Pass@k Actually Measures

Pass@k is the probability that at least one of k sampled completions solves the problem. It is not an accuracy figure and it is not a count of successes. Understanding how it is computed changes how you read every code-model announcement.

The Unbiased Pass@k Estimator, in Plain Terms

The obvious way to estimate pass@k is to take your measured pass@1 rate and compute 1 − (1 − p)^k. The Codex authors avoid this because it is a biased estimator, and use instead: generate n samples per problem where n ≥ k, count c correct ones, then compute 1 − C(n − c, k) / C(n, k) (arXiv:2107.03374).

Read that formula in words and it is simple. Out of all the ways to draw k samples from your n, what fraction of those draws contain zero correct answers? Subtract that from one. The original paper used n = 200 and k ≤ 100, which is a lot of generation per problem.

The practical implication is about reproducibility. Two labs reporting pass@10 can be doing genuinely different things if one sampled 200 completions and used the estimator while the other sampled exactly 10 and counted. The number looks comparable and is not. This is the same class of problem covered in what an eval harness actually controls.

Pass@1 vs Pass@10 vs Pass@100

Pass@1 is the honest measure of first-try reliability. It maps to how people actually use a model in an IDE: you get one completion and you either use it or you rewrite it. It is the hardest of the three and the most relevant to daily developer experience.

Pass@10 and pass@100 measure something closer to search. Given many attempts, does a correct solution exist somewhere in the model’s output distribution? That matters if you have a verifier that can pick the right one, such as a test suite the agent can run itself.

Without a verifier, high pass@100 is close to meaningless in production. You cannot ship “one of these hundred functions is correct” to a user. The gap between a model’s pass@1 and pass@100 tells you how much of its apparent capability depends on filtering you have to build yourself.

What Pass@k Does Not Prove

A passing HumanEval score proves that generated code ran and satisfied a small set of assertions. Everything past that is inference. Three specific problems make a high score much weaker evidence than it appears.

Contamination: When the Test Is Already in the Training Data

HumanEval was written by hand in 2021 to stay out of training data. It did not stay out. A study of leakage in code generation evaluation sets found that every HumanEval prompt appears at least 43 times on GitHub, with a median of 99 hits (arXiv:2407.07565).

Separate work quantifying contamination found substantial overlap between popular code generation benchmarks and open training corpora, with models performing measurably better on the contaminated subsets (arXiv:2403.04811). That is the shape of the problem: the score rises, and part of the rise is recall rather than reasoning.

You cannot cleanly separate the two from the outside. When a model trained on a 2024 web crawl scores 95% on a 2021 benchmark whose solutions have been reposted hundreds of times, no amount of interpretation tells you exactly how much of that is genuine synthesis.

Weak Test Suites: EvalPlus and Subtly Wrong Code

Averaging 7.7 tests per problem is thin coverage. The EvalPlus team tested that directly by extending HumanEval’s test cases by 80x, producing HumanEval+. The extra tests caught significant amounts of previously undetected wrong code, reducing pass@k by up to 19.3–28.9% (arXiv:2305.01210).

Worse than the drop is the reordering. The same paper found that test insufficiency causes mis-ranking, with WizardCoder-CodeLlama and Phind-CodeLlama beating ChatGPT on HumanEval+ despite trailing it on the original. The leaderboard order was an artifact of weak tests.

The effect is still visible on the live EvalPlus leaderboard. Across the 125 models listed there with both scores, the average drop from HumanEval to HumanEval+ is 5.0 points, and individual gaps run much wider. Gemini 1.5 Pro 002 falls from 89.0 to 79.3, and the top entries on that board drop from 96.3 to 89.0.

Single-Function Scope: No Repository, No Context

Every HumanEval problem is one function with no external dependencies. There is no second file, no import from your internal package, no config to read, no existing convention to match. The model is never asked to find where the change belongs.

That skips most of what makes real coding work hard. Locating the right file, understanding why the current code is written the way it is, and not breaking three other call sites are the parts that consume engineering time. A benchmark of standalone functions cannot see any of it.

The scope limit also hides an entire class of failure. A function that passes its tests can still leak a secret into a log line, swallow an exception, or add a dependency the project does not want. None of those show up as a failed assertion, so pass@k records them as success.

Is HumanEval Still Relevant in 2026?

Partly. It is a poor instrument for separating frontier models and a reasonable one for the rest of the field. Which of those you are doing decides whether the score is worth reading.

Frontier Saturation: Everyone Clusters at the Ceiling

Top-scoring systems now sit near the top of the scale, and the remaining gaps between them are small enough that sampling variance and prompt formatting can account for a point or two. On the EvalPlus leaderboard, the highest base HumanEval entries reach 96.3 pass@1, and that board has not tracked the newest frontier releases, which if anything understates today’s ceiling. There is not much headroom left to measure.

Compare that to the 28.8% Codex-12B managed in 2021 and the trajectory is obvious. A benchmark that separated models by fifty points now separates them by two. When your measuring instrument’s spread collapses to roughly its own noise floor, it has stopped being a comparison tool.

Treat any frontier HumanEval figure as a floor check, not a ranking. It confirms a model has not regressed at basic Python function synthesis. It does not tell you which of two models to build on.

Where It Still Gives Signal

Below the frontier the picture is different. Smaller and open-weight models spread across a wide range on the same board, from 2.4 for StableLM-7B up to 92.1 for Qwen2.5-Coder-32B-Instruct. That spread is real signal, and it is cheap to obtain since HumanEval runs fast and deterministically.

It is also useful as a regression check. If you are fine-tuning, quantizing, or distilling a model, a HumanEval run before and after tells you whether you broke basic code generation. That is a narrow question, and narrow questions are exactly what this benchmark answers well.

Just run HumanEval+ rather than the original when you do. The extra tests cost almost nothing and remove the most obvious way to pass with wrong code.

One caveat worth stating plainly. A small model that scores well on HumanEval has still only demonstrated basic Python function synthesis. It has not shown it can hold a long context, call tools correctly, or recover from its own mistakes. Read the score as a gate, not a recommendation.

What to Use Instead, or Alongside It

No single benchmark covers code generation. The practical approach is a small set with different failure modes, plus evaluation on your own traffic.

What Each One Adds

LiveCodeBench collects problems from LeetCode, AtCoder, and Codeforces with timestamps, so you can score a model only on problems published after its training cutoff, and the set keeps growing with each new contest. Its authors used that design across 52 models to surface evidence of overfitting to existing benchmarks.

SWE-bench moves the unit of work from a function to a pull request. It gives a model a real GitHub issue and a repository and asks for a patch that makes the project’s own tests pass. At publication, the best system resolved 1.96% of issues, which is the clearest possible statement of how different this task is from HumanEval. We covered the mechanics in the SWE-bench evaluation harness.

BigCodeBench targets the middle ground: 1,140 tasks that require composing calls across 139 libraries in seven domains, with about 5.6 tests each at 99% average branch coverage. Its authors report leading models reaching up to 60% against human performance of 97%, so unlike HumanEval it still has room to discriminate.

Where Static Benchmarks Stop

Every benchmark above shares one limit: the tasks are not yours. They use public repositories, public problems, and a scoring definition someone else chose. None of them run your prompts, your context window, your tools, or your users’ actual requests.

That is the handoff point to production evaluation. Once a model is inside your system, the questions change to whether it completed the task the user asked for, whether it followed your instructions, whether the output parses, and how often it fails in ways no test anticipated. Benchmarks and production evals answer different questions, and confusing the two is how a 96% score turns into a support backlog.

Future AGI

Future AGI does not ship a HumanEval replica or any code-benchmark leaderboard product. If you want a HumanEval number, run the official harness or EvalPlus. What the platform covers is the layer after the benchmark, once generated code is running inside a real system.

For scoring code output, the built-in library includes Code & Output Validation Checks, Contains Code, Is JSON, Instruction Adherence, and Task Completion, and you can define your own custom evals when none of those match your definition of correct (docs). Those run against your outputs rather than 164 fixed problems.

Simulation covers the gap between a single function and a working agent. You define agent configurations, customer scenarios, and personas, then run those interactions before launch, with results scored automatically (docs). That gets closer to multi-step coding-agent behavior than a standalone function ever will.

Observe records each production request as a trace built from nested spans, with sessions grouping a full conversation (docs). Error Feed then clusters, scores, and triages failures in those traces automatically, grouping fifty traces with one root cause into a single issue instead of fifty alerts (docs).

This is custom evals plus production tracing, aimed at the failures a static single-function pass/fail score structurally cannot see. It is not a competitor to HumanEval. You can run a fixed test set through it and get aggregate scores to compare two models or two prompt versions, which is the before-and-after check the section above argues for. What you do not get is a public, industry-comparable leaderboard number to put on a slide.

Conclusion

Pass@k tells you a model can solve isolated, contamination-exposed problems under one particular framing, scored by tests thin enough that EvalPlus knocked up to 28.9% off by adding more. That is a real result and a narrow one.

The score cannot tell you whether the model writes secure code, matches your conventions, navigates your repository, or completes the task a user actually asked for. Those failures live outside what 164 standalone functions can measure.

So use it as one input. Run HumanEval+ over the original, pair it with a time-windowed benchmark and a repository-level one, and then measure the thing you actually care about on your own traffic. The full set of LLM evaluation metrics is a good place to work out what that looks like for your system.