LLM Evaluation Metrics in 2026 and How to Choose Them
LLM evaluation metrics explained: exact match, BLEU, ROUGE, BERTScore, LLM-as-judge, groundedness, safety, latency and cost, and which ones to gate in CI.
Table of Contents
LLM evaluation metrics are the scores you use to decide whether a model’s output is good enough to ship. Almost all of them come from three families: deterministic checks, reference-based similarity scores and LLM-as-judge rubrics. Vendor catalogs list dozens of names, but most are the same few mechanisms pointed at a different failure mode.
TL;DR
| Layer | What it is | Count that matters |
|---|---|---|
| Primitive families | The mechanism the metric uses to score | 3 |
| Rubrics that ship in production | The named jobs the rubric does | 8 |
| Everything else | Permutations of the above | Skip until a real failure demands one |
The three primitive families
Nearly every LLM eval metric is built from one of three primitives or a stack of them. Learn the three and the catalog becomes a lookup table. Each entry below covers what the primitive is good at and where it breaks.
Deterministic. A function with no model in the loop: parse the response into JSON, validate against a schema, run a regex for a refusal phrasing, match a tool call against an expected signature. Deterministic checks are sub-10 ms, free, and never drift. They are also the wrong tool for “is this helpful,” so use them for closed-form questions where the answer is provably right or wrong against a rule.
Reference-based similarity. Score the candidate against a gold reference: BLEU, ROUGE and METEOR count overlapping words and n-grams, while BERTScore matches tokens through contextual embeddings and tolerates more paraphrase. All of them measure resemblance to the reference rather than correctness, so a confidently wrong answer that shares vocabulary still scores high. See What is BLEU, ROUGE, and BERTScore? for the full breakdown.
LLM-as-judge. A capable model reads the rubric, the candidate response and the context, then returns a score; G-Eval formalized the pattern in 2023. The judge is the only general-purpose primitive for rubrics that require reasoning (faithfulness, refusal calibration, role adherence, helpfulness), and it is also the most expensive primitive and the one most prone to bias. See the LLM-as-judge guide for the calibration pattern.
Classifier-backed metrics such as LlamaGuard 3, Qwen3Guard, Granite Guardian, WildGuard and ShieldGemma are the cheap judge: a small model fine-tuned for one safety taxonomy (the versions Future AGI runs locally span 0.6B to 8B parameters). Future AGI’s Guardrails class runs them, and its docs list the local ones at roughly 100 ms to 1 s per check.
The skill is matching the question to the cheapest primitive that answers it honestly. Running a frontier judge on a binary toxicity decision costs frontier-model prices for a call that QWEN3GUARD_0_6B settles locally in about 100 ms.
Classic text metrics and what each one measures
Classic text metrics, except perplexity, need a reference answer, so they fit offline test sets better than live traffic. Future AGI ships BLEU, ROUGE, Levenshtein similarity and embedding similarity as built-in evals, so this layer runs in the same suite as the rubrics below.
| Metric | What it measures |
|---|---|
| Exact match | Share of predictions that match a reference answer exactly |
| Token F1 | Average word overlap between prediction and reference |
| Perplexity | Exponentiated average negative log-likelihood of text under the model; lower is better |
The eight rubrics that matter in production
Start with these eight rubrics before anything else, and add specialized rubrics only when a real failure cluster forces you. The first six score output quality and safety; the last two, latency and cost, score whether a correct answer is fast and cheap enough to ship.
1. Groundedness asks whether the answer only asserts claims supported by the retrieved context. It is the RAG and QA core, and it catches hallucination against perfectly good retrieval. Implementation: a judge primitive against the context, which Future AGI ships as Groundedness.
2. Refusal asks whether the system refused what it should refuse, and answered what it should answer. Over-refusal of benign requests is as much a failure as under-refusal of harmful ones. Score both sides against the policy: Future AGI’s AnswerRefusal covers harmful requests, and a custom engine="llm" judge in evaluate() covers over-refusal.
3. Factual Accuracy is the final ground-truth check when you have one; when you don’t, the judge scores against an authoritative rubric. Distinct from Groundedness: Groundedness checks “did the model stick to the retrieved context,” Factual Accuracy checks “is the claim actually true in the world.” Future AGI’s FactualAccuracy scores the output against the ground-truth facts you pass in as context.
4. Toxicity is multi-axis scoring across insult, threat, identity attack, sexual, and violent content, scored by a judge or a classifier-backed guard model (LlamaGuard 3, ShieldGemma, Turing Flash). It is both a CI rubric and a runtime guardrail. Future AGI’s Guardrails class runs those classifiers at runtime, and its Toxicity eval is a pass/fail judge that flags hate speech, threats and abusive text.
5. PII / Data Privacy Compliance detects personal identifiers in inputs and outputs, optionally redacts them, and logs the policy decision. Mix of deterministic (regex packs for SSN, credit card, phone) and classifier (named-entity recognition for free-form PII). Future AGI ships detection as PII and GDPR and HIPAA compliance checks as DataPrivacyCompliance.
6. Tone asks whether the assistant stayed in its declared persona, register, and brand voice. Score it with a judge primitive against a tone rubric; this one is easy to under-spec, and teams tend to find the gap through a customer-support escalation. Pin the rubric to brand-voice exemplars rather than asking the judge for “professional tone” in the abstract.
7. Latency matters because a correct answer at 30 seconds is a product failure. Track p50 and p95 per route and gate p95 against an absolute floor in CI.
8. Cost breaks unit economics when a correct answer costs $0.50 per request. Track per-request cost and cost per resolved task, and gate cost per resolved task as a CI metric.
That is the list: eight rubrics, three primitives. Everything else in a vendor’s taxonomy, however long the list runs, is a permutation: ChunkAttribution is a pass/fail context-usage check, IsHarmfulAdvice narrows harm detection to advice a user might act on, and LLMFunctionCalling is the judge primitive pointed at a single tool call. Useful permutations exist, and you add them once one of the eight shows a gap.
The rubrics that show up in specific shapes
A few rubrics earn their slot only on specific application shapes. That means eight rubrics always, three to five more by shape, then stop.
RAG-specific. ContextRelevance (do retrieved chunks address the question?), ContextAdherence (does the answer only use what was retrieved?), Completeness (does the answer cover every part of the question?), ChunkAttribution (did the answer use the retrieved chunks at all?), ChunkUtilization (how much does the answer draw on them?). The RAG evaluation metrics guide covers the per-stage pattern.
Agent-specific. LLMFunctionCalling (right tool, right parameters), TaskCompletion (did the response accomplish what the user asked for?), PromptInjection (does an input try to make the model ignore its instructions?). Run them on every step of the trace as well as the final response; they need the full span tree, which is why agent eval and tracing are the same system in 2026.
For a per-metric reference covering tool selection, argument correctness, plan adherence, recovery and handoffs, read AI agent evaluation metrics.
Conversation-specific. ConversationResolution (did the dialogue reach a satisfactory end state?), Knowledge Retention (did the agent carry facts across turns?), Outcome Label (resolved, filed, booked, refunded?). Per-turn evals on multi-turn agents produce false confidence; add conversation-level rubrics. See Multi-turn LLM Evaluation.
Format-specific. IsJson, ContainsValidLink, IsEmail, length bounds and schema match are deterministic checks that run in microseconds, so run them first on every trace. A malformed response fails before any judge needs to read it, and the check returns the same answer every time.
Pick by application shape
| Shape | Core rubrics (CI gate every PR) | Add when |
|---|---|---|
| Chat assistant | Refusal, Tone, TaskCompletion + Toxicity, PII + Latency, Cost | Multi-turn → ConversationResolution |
| RAG / KB / support | Groundedness, ContextRelevance, Completeness + Toxicity, PII + Latency, Cost | Have ground truth → FactualAccuracy |
| Tool-using agent | LLMFunctionCalling, TaskCompletion, AnswerRefusal + PromptInjection + Latency, Cost | Touches external state → permission-scope check |
| Code generation | Pass-at-1 on fixtures, static analysis, security patterns + Latency, Cost | Security-sensitive → secret-scanning rubric |
| Safety-critical (health, finance, legal) | Domain harm rubric + full Scanner pack as inline guardrails + Refusal | Regulated → domain-specific compliance template |
Every shape adds the safety triad (Toxicity, PII, PromptInjection) and the latency-cost pair. The mistake is starting from the catalog and picking by name. Start from the failure modes that hurt your users and pick by job. A short list calibrated against human labels catches more regressions than a long one nobody reviews.
The CI gate composition
A metric you run once is a slide in a deck; a metric you run on every PR is a quality gate. The gate fails the pull request when a pinned rubric drops below its threshold, and it has four parts:
- Versioned dataset. 50-100 examples per route, sampled from production, biased toward the hardest 10 percent. Refreshed weekly by promoting failing production traces.
- Pinned rubrics. A fixed set tied to the application shape, with
(judge_model_id, rubric_version, prompt_template_hash)pinned. A vendor swap on the judge is a deliberate eval-suite migration, not a config change. - Per-route thresholds. Fail the PR if any rubric drops more than two points from the trailing 7-day baseline, or falls below an agreed absolute floor. 0.75 on Groundedness, 0.85 on TaskCompletion are reasonable starting defaults to tune.
- Cascade-aware runner. The runner that executes the rubric uses the cascade (deterministic first, classifier next, judge last), so the CI bill is proportional to ambiguity, not to dataset size.
Future AGI’s evaluate() runs these rubrics from a CI script that exits non-zero when any case fails. Put local checks such as is_json in front of Turing judges like groundedness, answer_refusal and toxicity, so only the cases that pass them reach a judge. The CI/CD eval pipeline recipe wires it into GitHub Actions.
The CI gate is a regression suite, not a leaderboard. The score that matters is the diff against last week’s baseline, not the absolute number on a model card. The full buildout is in Build LLM evaluation framework from scratch.
The hybrid cascade that makes per-trace eval affordable
No single primitive wins: deterministic checks miss everything semantic, similarity scores miss everything that needs reasoning, and a judge is too expensive to run on every production trace. The production answer is a cascade that sends each trace to the cheapest layer that can settle it.
Future AGI’s evaluate() function runs local metrics, cloud Turing evals and LLM judges, and augment=True runs a local heuristic first, then hands its score and reasoning to an LLM judge. That flag refines one eval; the cascade is a routing pattern you build around it, with three layers:
- Deterministic layer. Scanners, regex and schema checks run locally in under 10 ms.
- Classifier layer. A small guard model such as
QWEN3GUARD_0_6Bscreens what passes. - LLM-as-judge layer. A pinned rubric judge scores only what is left.

Because traces that settle in the local layers never reach the judge, the judge bill tracks the share of ambiguous traces instead of total volume. That is what makes scoring every production span affordable, instead of sampling 1 percent and hoping the missed 99 percent stay clean.
The cascade is the operational answer to the “LLM eval is too expensive” complaint. The whole stack does not have to be cheap; the average has to be cheap, and the cascade gets the average there. See Deterministic vs LLM-judge evals for the deeper treatment.
Common mistakes
- Over-rely on BLEU and ROUGE. They reward surface overlap, not meaning. Use them as a cheap signal layer, not the primary quality metric.
- Run only LLM-judge. The bill grows faster than the inference bill. Cascade through deterministic and classifier first.
- Pick the highest-scoring judge. Judge-as-marketing. Pick the one that agrees with human labels.
- Score the response, ignore the trace. Agent failures often live in the tool call or retrieval step. Use agent-specific rubrics and capture the trace.
- Skip calibration. Every classifier and judge needs periodic agreement checks against human labels. Without calibration you have a number, not a metric.
- Frozen dataset. A static dataset stops being a regression suite the moment production drifts. Promote failing traces weekly.
- Treat safety as a separate stack. Safety rubrics belong in the same Evaluator call as quality rubrics. One pipeline, multiple rubrics, one cascade.
- Latency and cost as graphs, not gates. A correct answer at 30 seconds is a product failure. Gate it in CI.
How Future AGI ships every family
Future AGI covers all three families in one stack. Start with the open-source SDK, then run the same evals on the platform against datasets and live traces. Every eval returns a score (pass/fail, a number or a category) and, when an evaluator model is involved, a plain-language reason.
- agent-learning-kit (Apache-2.0).
pip install agent-learning-kit. Local metrics likebleu_scoreneed no API key, Turing models score templates likegroundedness, andengine="llm"runs your own judge on any LiteLLM model. - Future AGI platform (Apache-2.0 core, self-hostable). Attach the same evals to Observe traces and score datasets in bulk.
- Error Feed. Groups traces that failed the same way into one issue, can link it to a Linear ticket, and writes a root cause analysis with a proposed fix.
- traceAI (Apache 2.0). OpenTelemetry instrumentation for 50+ AI frameworks in Python, TypeScript, Java and C#;
EvalTagrubrics passed toregister()score live spans.

The hosted Agent Command Center is an OpenAI-compatible gateway (100+ providers, 15 routing strategies) whose x-agentcc-cost and x-agentcc-latency-ms headers report per-request numbers for the Latency and Cost rubrics. Future AGI is SOC 2 Type II, ISO 27001, GDPR and CCPA certified, and HIPAA compliant.
From dozens of metric names to one short list
A team staring at a vendor catalog with dozens of metric names needs a short list it can run. Three primitives, eight rubrics gated on every pull request, a few rubrics for your app’s shape, and a cascade that sends each trace to the cheapest layer that can settle it cover the job.
Ready to score your own traffic? Run pip install agent-learning-kit, sign up for Future AGI for the API keys that cloud evals need, then score your last fifty production traces with groundedness and answer_refusal.
Frequently Asked Questions
What are LLM evaluation metrics?
How do you measure LLM accuracy?
What is LLM as a judge evaluation?
What are the most common LLM benchmarks?
What is time to first token in LLM metrics?
The pillar playbook for LLM evaluation in 2026: dataset, metrics, judge, CI gate, production observation, closed loop from failing trace to regression.
Benchmarks tell you which model is smartest. Metrics tell you if your system works. 2026 guide: benchmark map, metric catalog, CI gate, rubric.
Schema, regex, exact match, BLEU/ROUGE, citation-validity. Where deterministic LLM eval metrics catch 30-60 percent of failures before a judge fires.