LLM Leaderboard Explained in 2026: Arena, MMLU, MMMU, GPQA, SWE-bench, and How to Read the Charts

LLM leaderboard explained in 2026: TL;DR

Pick the leaderboard that matches your use case. For agent reliability and coding, watch SWE-bench Verified. For reasoning, watch GPQA Diamond and AIME. For perceived quality on free-form prompts, watch LMArena. For open-weight model selection, watch the Artificial Analysis Intelligence Index, because the Hugging Face Open LLM Leaderboard that used to hold that job has been retired.

Why leaderboards still matter

A leaderboard does three things well:

1. Coarse model shortlist. If two models score within five percent on the benchmark closest to your use case, they are both worth a real evaluation. If one is 30 points behind, it is probably not.
2. Vendor accountability. Frontier labs publish benchmark numbers in launch blog posts; the community then verifies on independent harnesses (lm-eval, HELM, simple-evals).
3. Progress tracking. The shape of the leaderboard frontier shows where capability is and is not advancing. Saturation on MMLU and HumanEval, plus rapid gains on GPQA Diamond and SWE-bench Verified, tell you where research effort is going.

What leaderboards cannot do: predict how a model will behave on your specific prompts, with your tool schema, in your language, on your latency and cost budget. That is what evaluation on your own data is for. For the deeper read on what the public scores can and cannot tell you, see the state of LLM benchmarking.

The benchmarks that matter in 2026

LMArena (Chatbot Arena)

Arena — formerly LMArena, and before that LMSYS Chatbot Arena — is the open-source platform the whole category grew out of. The lmarena.ai domain now redirects to arena.ai. Humans see two anonymized model responses to the same prompt and pick the better one; an Elo rating is computed over millions of comparisons. The platform now ships:

• Arena for general chat.
• Code Arena for code generation tasks.
• Vision Arena for image-grounded prompts.
• Hard Prompts subsets that filter on harder queries.
• Multi-turn arena for multi-step conversations.

LMArena rewards instruction following, helpfulness, and style alignment. It penalizes hallucinations, refusals, and verbose responses humans find annoying. It is the best single signal for “would users prefer this model” but is less useful for measuring factuality, code correctness, or math.

MMLU and MMLU-Pro

MMLU is a 57-subject multiple-choice exam covering STEM, humanities, social science, and professional topics (law, medicine). Frontier models cluster near the top of it, which is why MMLU is treated as largely saturated and has been displaced by harder sets. Check a current leaderboard for per-model figures rather than quoting a range from an article.

MMLU-Pro is the harder successor: more answer options, more reasoning-heavy questions, less rote recall. Frontier models have not saturated it the way they saturated MMLU, so it still carries signal.

MMMU

MMMU is the multimodal counterpart to MMLU: college-level questions across 30 subjects with both text and images (charts, diagrams, medical images, screenshots). The MMMU-Pro variant removes shortcuts and is the one worth watching, since the base set is closer to saturation.

GPQA Diamond

GPQA (Graduate-Level Google-Proof Q&A) is a benchmark of 448 questions in physics, biology, and chemistry written by PhDs and validated to resist web lookup. The Diamond split is the 198 hardest questions. Human PhDs in the relevant subject score around 65 percent; non-experts with internet access score around 34 percent.

In August 2026 GPQA Diamond is the most-watched hard-reasoning benchmark. Frontier models commonly report high GPQA Diamond scores in launch posts; verify the specific numbers on each model’s official benchmark card or scaling-report blog.

SWE-bench Verified

SWE-bench presents a model with a real GitHub issue plus the repository at the relevant commit and asks for a patch that passes the project’s tests. SWE-bench Verified is the 500-task subset OpenAI manually filtered for solvability and unambiguous specifications.

This is the closest 2026 has to “can this model actually do production engineering work.” Claude Opus 5 and Fable 5 with the Claude Code harness, GPT-5.6 Sol with the OpenAI Agents SDK, and agent products such as Devin and Cursor trade leadership; verify the current SWE-bench Verified scores on the official leaderboard at swebench.com before quoting numbers.

HumanEval and HumanEval+

HumanEval is a 164-task Python code generation benchmark where the model completes a function from a docstring. HumanEval+ is the contamination-resistant successor with additional tests. Frontier models have effectively saturated both, which is why launch posts rarely lead with them any more. Use it as a sanity check; use SWE-bench Verified for serious code-ability measurement.

AIME 2025 and HMMT

AIME is the American Invitational Mathematics Examination; HMMT is the Harvard-MIT Mathematics Tournament. Both produce 15-problem competition sets per year. Frontier reasoning models report high AIME scores when allowed extended thinking and best-of-k sampling, versus 10 to 40 percent for non-thinking base models.

HELM

HELM from Stanford CRFM is a “holistic” evaluation framework that scores models on a large matrix of scenarios (NarrativeQA, BoolQ, NaturalQuestions, MS MARCO, MMLU, GSM8K, math, code, biases, toxicity, calibration) and metrics (accuracy, calibration, robustness, fairness, bias, toxicity, efficiency). HELM is slower-moving and research-grade. It is the best holistic single source if you want to compare across many dimensions in one place.

The Hugging Face Open LLM Leaderboard is retired

For two years this was the answer to “which open-weight model is best”. It is not any more, and if you learned to check it by habit, that habit now points at a static snapshot.

The history in one line: v1 ran on six benchmarks (ARC, HellaSwag, MMLU, TruthfulQA, Winogrande, GSM8K) and was archived in June 2024. v2 replaced it with a harder set built to resist saturation (MMLU-Pro, GPQA, MuSR, IFEval, BBH, MATH Lvl 5). Hugging Face then retired the leaderboard outright, and the reason they gave is worth reading if you take benchmarks seriously: the leaderboard was “slowly becoming obsolete”, and they judged that it “could encourage people to hill climb irrelevant directions in the field”. They chose to stop it before that happened rather than after.

That is a rare and honest thing for a leaderboard operator to say, and it is the single best argument in this article for why you should not outsource model selection to a public ranking. The benchmarks a leaderboard fixes in place become the targets the whole field optimises against, and the moment those targets stop correlating with useful behaviour, the leaderboard is actively steering people wrong.

What to use instead, depending on what you were using it for:

• Cross-model comparison, open and closed together: the Artificial Analysis Intelligence Index, below.
• A specific capability: one of the 200-plus community leaderboards now hosted on Hugging Face, discoverable through the OpenEvals/find-a-leaderboard space. These are narrower and better maintained than one general ranking could be.
• Reproducing scores yourself: the underlying EleutherAI LM Evaluation Harness is still published and maintained, so the methodology outlived the leaderboard.

Artificial Analysis Intelligence Index

Artificial Analysis has taken over most of the job the Open LLM Leaderboard used to do, with one important difference: it runs the evaluations itself rather than accepting provider-reported numbers. That independence matters more every year, as launch-day benchmark cards get more selective about which numbers they print.

The index is a single score combining nine evaluations. Version 4.1 uses GDPval-AA v2, τ³-Banking, Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond, CritPt, AA-Omniscience, and AA-LCR.

Read that list closely, because the composition is the story. Terminal-Bench, τ³-Banking and GDPval are agentic and task-completion evaluations, not question-answering sets, and v4.1 explicitly reweighted the index toward agentic workloads. The industry’s headline measure of “intelligence” has quietly stopped being about answering hard questions and started being about finishing multi-step work. That is the same shift the agentic benchmarks section below describes, now reflected in the index most buyers cite.

Rankings here move week to week, and the same model can place differently depending on its reasoning-effort setting, so check the live leaderboard rather than quoting a position from any article, including this one.

Agentic benchmarks

The newer wave of agent-focused benchmarks measures end-to-end task completion, not single-response quality:

• GAIA for general-assistant tasks across reasoning, multimodal, and web browsing.
• OSWorld for desktop automation in a real OS environment.
• TAU-bench for customer-service-style tool-use scenarios.
• WebArena and VisualWebArena for web-navigation tasks.
• Aider polyglot for code-editing across multiple languages.

These benchmarks are where the agentic-model narratives in 2026 are being decided. Watch them alongside SWE-bench Verified for production agent decisions.

Current state of the top of the leaderboard in August 2026

The table below is an illustrative snapshot of the August 2026 frontier across closed and open-weight model families. Exact ordering on any specific benchmark shifts week by week; verify live rankings on the source leaderboards (LMArena, SWE-bench, Artificial Analysis) before making a model decision.

Note how much of that table turned over since the previous version of this article, which had GPT-5, Claude Opus 4.x and Gemini 2.5 Pro at the frontier only a few months ago. That churn rate is the practical argument for treating any leaderboard snapshot in an article, including this one, as a starting point rather than an answer.

For the live picture see our best LLMs in July 2026 writeup, which tracks the current frontier across closed and open weights with current pricing and capability notes.

Beyond raw accuracy: latency, cost, and reliability

Leaderboards rank capability. Production decisions weigh capability against the other axes:

Latency

A 90-percent-MMLU model that takes 30 seconds to respond is unusable for a chatbot. Watch:

• Time to first token (TTFT).
• Tokens per second (output throughput).
• End-to-end p50, p95, p99 latency on your prompt distribution.
• Reasoning overhead when using extended-thinking modes.

Cost

Frontier closed-model pricing in August 2026 spans a wide range, from sub-cent per million input tokens for small open-weights on rented GPUs up to roughly $25 to $30 per million output tokens at the top of the Anthropic and OpenAI lines. Cost per successful task is what to optimize; benchmark accuracy alone hides cases where a cheaper model is “good enough” for 80 percent of traffic.

Reliability

• Instruction following (IFEval).
• Hallucination rate on your domain.
• JSON-schema validity for structured outputs.
• Tool-call correctness for agentic stacks.
• Refusal rate and false-positive safety blocks.

These rarely make it onto public leaderboards but they are what makes or breaks a production deployment.

Why leaderboards disagree with your own results

Three common reasons:

Distribution mismatch

A model that scores 88 percent on MMLU may score 60 percent on your customer-support classification dataset because the prompt style, the input length, the language, and the answer format are all different.

Benchmark contamination

Frontier models train on web data that includes benchmark questions. Despite contamination-detection work (perplexity filters, n-gram overlap checks, paraphrase-based evals), some benchmarks leak into pretraining and inflate reported scores. GPQA, AIME 2025, MMLU-Pro, and SWE-bench Verified were designed with this in mind; older benchmarks (HumanEval, MMLU, GSM8K) are more affected.

Single-response vs production behavior

A benchmark scores one response. Production agents do many things: retrieve, plan, call tools, observe results, retry, hand off. A model that wins on single-response GPQA may lose on the same questions wrapped in an agent loop because it cannot follow tool schemas or recover from bad observations.

How to evaluate an LLM on your own data

The disciplined approach in August 2026:

1. Build a dataset of 100 to 500 representative inputs from production, anonymized. Cover the long tail, not just the happy path.
2. Define metrics per input:
   • Exact match for classification
   • JSON-schema validity for structured outputs
   • Regex or fuzzy match for extraction
   • LLM-judge (faithfulness, instruction following, helpfulness, hallucination) for open-ended quality
3. Run every candidate model through the same dataset with the same evaluators.
4. Aggregate by overall score, by intent or domain, by latency band, and by cost band. Tail behavior matters as much as the mean.
5. Repeat regularly. Vendors update closed models silently; pinning weights for open-weight models is the only way to lock baseline performance.

Future AGI’s Apache 2.0 Agent Learning Kit and Future AGI cloud evals API ship faithfulness, groundedness, instruction following, hallucination, tone, completeness, and tool-call correctness evaluators that work consistently across OpenAI, Anthropic, Google, Mistral, and self-hosted models. The Apache 2.0 traceAI library emits OpenTelemetry spans that bind every model call to the eval run that scored it.

from fi.evals import evaluate

# Illustrative snippet: replace the placeholders with your real values.
candidate_answer = "<the candidate model's response>"
retrieved_passage = "<the passage the model was supposed to ground in>"

# Score whether the candidate model's answer is faithful to the retrieved context.
result = evaluate(
    "faithfulness",
    output=candidate_answer,
    context=retrieved_passage,
)
print(result.score, result.reason)

from fi.evals.metrics import CustomLLMJudge
from fi.evals.llm import LiteLLMProvider

judge = CustomLLMJudge(
    provider=LiteLLMProvider(),
    model="gpt-5",
    name="domain-specific-grader",
    prompt="Score the answer 1-5 on relevance to the customer question.",
)
score = judge(input="Where is my order?", output=candidate_answer)

For BYOK gateway routing, prompt versioning, and live guardrails across all candidates, the Future AGI Agent Command Center sits in front of the major providers with one consistent API and environment variables FI_API_KEY and FI_SECRET_KEY.

Ethical considerations on modern leaderboards

Public leaderboards increasingly score more than raw capability:

• Bias detection. CrowS-Pairs, StereoSet, and BBQ measure demographic-group bias in completions.
• Toxicity. Real Toxicity Prompts and HarmBench measure output safety.
• Refusal calibration. XSTest and OR-Bench measure whether models refuse safe prompts.
• Privacy. Membership-inference and PII-extraction evals measure training-data leakage.
• Energy and efficiency. ML.energy and Green AI leaderboards track watt-hours per token.

Frontier model launch posts in 2026 include capability scores plus a safety scorecard; both are worth reading.

How LLM leaderboards shape the industry

Three effects worth naming:

• Model selection. Leaderboard scores remain the default first filter for shortlisting models. A buying decision that ignores leaderboards entirely usually misses important capability gaps; a buying decision that relies only on leaderboards usually ships the wrong model.
• Competitive pressure. Visible leaderboards push labs to invest in the benchmarks the field cares about, which advances the state of the art faster but also drives benchmark targeting.
• Standardization. Public evaluation harnesses (lm-eval-harness from EleutherAI, simple-evals from OpenAI, HELM from Stanford CRFM) reduce the noise in cross-lab comparison and give the community a path to reproducibility.

Bottom line

In August 2026, the leaderboard you should care about depends on what you are shipping. For agent reliability and code: SWE-bench Verified and Aider polyglot. For hard reasoning: GPQA Diamond and AIME. For perceived quality on real prompts: Arena. For open-weight model selection: the Artificial Analysis Intelligence Index, now that the Hugging Face Open LLM Leaderboard is retired. For multimodal: MMMU and MMMU-Pro. None of them substitute for evaluation on your own data, and that is the part most teams skip. Treat the leaderboard as the shortlist, then run a controlled evaluation with a consistent harness before deploying. For frontier model picks in August 2026 specifically, see our best LLMs guide and the deeper LLM benchmarking comparison.

Sources

• End of the Hugging Face Open LLM Leaderboard, with the maintainers’ stated reasoning: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard/discussions/1135
• Open LLM Leaderboard v1 archive page (June 2024) and its six benchmarks: https://huggingface.co/docs/leaderboards/en/open_llm_leaderboard/archive
• Artificial Analysis Intelligence Benchmarking methodology, including the v4.1 evaluation set: https://artificialanalysis.ai/methodology/intelligence-benchmarking
• GPQA paper, Rein et al. (arXiv:2311.12022): https://arxiv.org/abs/2311.12022
• SWE-bench Verified: https://www.swebench.com/verified.html
• Arena (formerly LMArena) leaderboard: https://arena.ai/