BLEU vs ROUGE vs BERTScore: Worked Examples and 2026 Use Cases

BLEU vs ROUGE comes down to precision versus recall. BLEU (Bilingual Evaluation Understudy) checks how much of a model’s output appears in a human reference and was built for machine translation, while ROUGE (Recall-Oriented Understudy for Gisting Evaluation) checks how much of the reference the output recovers and was built for summarization.

BERTScore vs ROUGE is a question of meaning versus exact words: BERTScore compares contextual embeddings, so it credits paraphrases that share few tokens. BLEU (2002), ROUGE (2004), and BERTScore (ICLR 2020) all still run in eval pipelines, and all three need a reference. This guide runs the same examples through each one with computed scores, shows where each breaks, and maps them against LLM-as-a-judge scoring.

TL;DR

All three need a reference. The 2026 production default for open-ended generation (RAG, dialog, creative) is rubric-bound LLM-judge scoring, which is reference-free. BLEU, ROUGE, and BERTScore remain useful for benchmark continuity, as cheap regression checks, and on tasks where references exist (translation, summarization with gold abstracts).

BLEU measures n-gram precision with a brevity penalty

BLEU (Bilingual Evaluation Understudy) was introduced by Papineni, Roukos, Ward, and Zhu in 2002 as a metric for automatic machine-translation evaluation. The authors pitched it as a quick, inexpensive, language-independent understudy to human judges that correlates highly with human evaluation.

How BLEU works

Given a candidate translation and one or more reference translations, BLEU computes:

1. Modified n-gram precision for n=1 through n=4. For each n, count how many candidate n-grams appear in any reference, capped at the maximum count in the references.
2. Geometric mean of the n-gram precisions, typically with equal weights (0.25 each for n=1 to 4).
3. Brevity penalty (BP): if candidate length c is less than reference length r, multiply by exp(1 - r/c). Otherwise BP is 1.

The final score is BP times the geometric mean of n-gram precisions, expressed as a value between 0 and 1 (or 0 to 100 in many publications). Google Cloud’s Translation docs write the same formula as the brevity penalty times the fourth root of the product of the four n-gram precisions.

Worked example

candidate: the cat sat on the mat
reference: the cat sat on the mat
BLEU-4: 1.00 (perfect match)

candidate: a cat is sitting on the mat
reference: the cat sat on the mat
1-gram precision: 4/7 (candidate tokens a, cat, is, sitting, on, the, mat -> modified matches cat, on, the, mat = 4; candidate has one "the", so it counts once)
2-gram precision: 2/6 (a cat, cat is, is sitting, sitting on, on the, the mat -> matches on the, the mat = 2)
3-gram precision: 1/5 (only "on the mat")
4-gram precision: 0/4 (no 4-gram match)
BP: 1.0 (candidate length 7 is greater than reference length 6, so no brevity penalty)
BLEU: 0 without smoothing; SacreBLEU sentence_bleu with default exponential smoothing gives 0.26

When BLEU breaks

• Creative writing and dialog. Many valid outputs exist; BLEU rewards copying the reference word for word.
• Single-reference tasks. When only one gold answer exists (most production tasks), BLEU under-rewards correct alternative phrasings.
• Short outputs. Brevity penalty dominates the score when the candidate is much shorter than the reference.
• Tokenization sensitivity. Post (2018) found differences as high as 1.8 BLEU points between commonly used BLEU configurations, mostly from different tokenization and normalization of the reference. Use SacreBLEU for reproducibility.
• Cross-lingual. BLEU is symbol-blind; high BLEU does not always mean meaningful translation in low-resource languages.

When BLEU still works

• WMT-style translation benchmarks with multiple high-quality references.
• Regression checks where you want a fast deterministic signal that nothing catastrophically broke.
• Continuity with two decades of published baselines.

ROUGE measures recall-oriented overlap for summarization

ROUGE (Recall-Oriented Understudy for Gisting Evaluation) was introduced by Chin-Yew Lin in 2004 for evaluating summaries. Where BLEU asks how much of the output appears in the reference, ROUGE asks how much of the reference appears in the output.

ROUGE variants

• ROUGE-N. N-gram recall: count of n-grams in the reference that appear in the candidate, divided by the total n-grams in the reference. ROUGE-1 is unigram, ROUGE-2 is bigram.
• ROUGE-L. Longest common subsequence between candidate and reference. Captures sentence-level structure.
• ROUGE-S. Skip-bigram: pairs of words in their order, with gaps allowed.
• ROUGE-W. Weighted longest common subsequence.

Modern reporting often uses ROUGE-1, ROUGE-2, and ROUGE-L F1 (precision and recall combined). The rouge-score package reports all three with precision, recall, and F-measure, plus an optional Porter stemmer for word forms like run and running.

Worked example

candidate: the cat sat on the mat happily
reference: a happy cat was sitting on the mat
ROUGE-1 recall: 4/8 (cat, on, the, mat from reference appear in candidate; a, happy, was, sitting do not) = 0.50
ROUGE-1 precision: 4/7 = 0.57
ROUGE-1 F1: 0.53
ROUGE-2 recall: 2/7 ("on the", "the mat") = 0.29
ROUGE-L: longest common subsequence is "cat on the mat" = 4 tokens; F1 ~ 0.53

When ROUGE breaks

• Abstractive summarization. A system that rephrases gets penalized; an extractive system that copies wins.
• Long outputs. ROUGE-L favors lexical overlap; coherence and structure are missed.
• Factuality. A summary can score high on ROUGE-N and contain factual errors.
• Multiple valid summaries. A single reference reflects one summary style; alternatives are under-credited.

When ROUGE still works

• CNN/DailyMail and XSum benchmark continuity.
• Cheap regression checks alongside other metrics.
• Coverage diagnostics where you want to know whether key phrases from the source made it into the summary.

BERTScore measures contextual embedding similarity

BERTScore was introduced by Zhang, Kishore, Wu, Weinberger, and Artzi at ICLR 2020. It replaced surface n-gram overlap with similarity in a learned embedding space. The paper tested it on outputs from 363 machine translation and image captioning systems.

How BERTScore works

1. Embed each token in the candidate using a pretrained transformer (roberta-large is the English default in the authors’ bert-score package, whose README now recommends microsoft/deberta-xlarge-mnli).
2. Embed each token in the reference the same way.
3. For each candidate token, find the maximum cosine similarity against any reference token. Average across candidate tokens (precision) and reference tokens (recall). F1 is the harmonic mean.
4. Optional: apply IDF weighting to down-weight common words.

BERTScore can range from -1 to 1, but with roberta-large it often lands in a narrow band around 0.85 to 0.95, and close paraphrases go higher. The authors’ bert-score package can rescale against a random-pair baseline, which widens the range without changing correlation with human judgment, so report which setting you used.

Worked example

candidate: the canine slept on the rug
reference: the dog rested on the carpet

Surface n-gram overlap (BLEU/ROUGE): low. The two share only "the" and "on the".
BERTScore F1 (RoBERTa-large): 0.97 raw, 0.82 rescaled with baseline, because canine ~ dog, slept ~ rested, rug ~ carpet in embedding space.

The example shows what BERTScore adds on paraphrase-rich tasks. BLEU-4 without smoothing scores this pair 0 because no 3-gram or 4-gram matches (SacreBLEU’s default smoothing lifts it to 0.18), and ROUGE-1 F1 is only 0.50. A human reads the two sentences as near-equivalent.

When BERTScore breaks

• Out-of-domain text. Biomedical, legal, or code text where the BERT-family base model was not trained: embeddings become noisy.
• Opposite meanings. Swapping one word for its opposite barely moves the score. In the revenue example below, the factually opposite candidate C gets the highest BERTScore of the three.
• Long-form generation. Token-level similarity does not capture global coherence.
• Reference dependency. BERTScore still needs a reference; production tasks without one cannot use it directly.

When BERTScore works

• Translation and summarization where references exist and paraphrase is common.
• Image captioning (tested alongside machine translation in the original paper).
• Cross-domain text where surface overlap is misleading but semantic equivalence is real.

BLEU vs ROUGE vs METEOR

METEOR (Banerjee and Lavie, 2005) was built to track human judgments better than BLEU. It matches unigrams by exact form, Porter stem, and WordNet synonym, scores precision and recall, and adds a fragmentation penalty when matches fall into scattered chunks. That lets it credit “running” against “run”, a pair BLEU counts as a miss and ROUGE matches only when its optional stemmer is on. All three still need a human reference.

Comparing the three on the same example

reference: the company announced a 12 percent revenue increase year over year
candidate A: the company reported a 12 percent rise in annual revenue
candidate B: revenue at the firm grew 12 percent compared to last year
candidate C: the company posted a 12 percent decline year over year (factually opposite)

BLEU (SacreBLEU sentence_bleu, default settings, 0 to 1 scale):
  A: 0.19
  B: 0.09
  C: 0.27 (highest, with the wrong meaning)

ROUGE-L F1 (rouge-score, no stemming):
  A: 0.57
  B: 0.36
  C: 0.76

BERTScore F1 (roberta-large, transformers 4.57, raw / rescaled with baseline):
  A: 0.96 / 0.78
  B: 0.91 / 0.44
  C: 0.97 / 0.80 (highest, although decline reverses increase)

Human judgment:
  A: correct, well-paraphrased
  B: correct, more abstracted
  C: WRONG (factually opposite)

All three metrics rank C, the factually opposite answer, first and B, the correct abstract paraphrase, last. BLEU and ROUGE reward C’s matching surface tokens, and BERTScore follows because decline and increase sit in similar contexts. None of the three catches factuality, the gap a reference-free LLM judge with a factuality rubric is built to fill.

How BLEU, ROUGE, and BERTScore correlate with human judgment by task

Future AGI’s LLM-as-Judge evals such as Groundedness and Detect Hallucination cover the factuality gap without a gold reference.

How to calculate BLEU score, ROUGE score, and BERTScore in Python

Three packages cover the classic metrics: sacrebleu for BLEU, Google’s rouge-score for ROUGE, and bert-score for BERTScore. The snippet reproduces candidate A’s numbers from the revenue example. Watch the argument order: sacrebleu takes the output first, while rouge-score takes the reference first. The BERTScore numbers come from transformers 4.57, and the install line pins transformers below version 5.

# pip install sacrebleu==2.6.0 rouge-score==0.1.2 bert-score==0.3.13 "transformers<5"
import sacrebleu
from rouge_score import rouge_scorer
from bert_score import score

ref = "the company announced a 12 percent revenue increase year over year"
hyp = "the company reported a 12 percent rise in annual revenue"

print(round(sacrebleu.sentence_bleu(hyp, [ref]).score, 1))  # 18.6 on a 0 to 100 scale
scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"])
print(round(scorer.score(ref, hyp)["rougeL"].fmeasure, 2))  # 0.57
P, R, F1 = score([hyp], [ref], lang="en")
print(round(F1.item(), 2))  # 0.96 with the default roberta-large

For a test set, call sacrebleu.corpus_bleu on all outputs at once instead of averaging sentence scores, since BLEU is a corpus-level metric. corpus_bleu takes the outputs first, then a list of reference streams, each holding one reference per output in the same order: sacrebleu.corpus_bleu(outputs, [refs]).

How to read BLEU and ROUGE scores

What is a good BLEU score?

Google Cloud’s Translation docs give a guide for corpus-level BLEU on a 0 to 100 scale: under 10 is almost useless, 10 to 19 makes the gist hard to get, 20 to 29 has a clear gist with significant grammatical errors, and 30 to 40 is understandable to good. Scores of 40 to 50 are high quality, 50 to 60 very high quality, and above 60 often better than human.

The same docs warn that BLEU performs poorly on individual sentences, which is why the sentence scores in this post look low even for correct answers. Compare BLEU only across runs that share a tokenizer and settings, which is the job SacreBLEU does.

What is a good ROUGE score?

Read ROUGE against a baseline on the same dataset instead of a fixed cutoff. Keep the package, variant, and stemming setting identical across runs, then check that your model beats a simple baseline such as the first sentences of the source. In the revenue example, candidate A, a correct paraphrase, scores 0.57 ROUGE-L F1 and the reversed claim 0.76, so a high ROUGE score never proves an output is right.

What replaced these metrics in 2026 production

For new production evals on open-ended LLM outputs, three approaches have largely displaced BLEU/ROUGE/BERTScore:

Rubric-bound LLM-judge

A judge prompt scores the output on explicit dimensions (factuality, completeness, fluency, helpfulness) and returns a reason with the score. G-Eval (Liu et al. 2023) used GPT-4 with chain-of-thought scoring and reported an average 0.514 Spearman correlation with human ratings on the SummEval summarization benchmark, ahead of all earlier methods it compared. Future AGI ships this pattern as its LLM-as-Judge eval type.

Pairwise preference

For ranking model variants, a pairwise judge (“which of A or B is better, and why”) fits better than an absolute score. Zheng et al. (2023) found strong judges like GPT-4 reach over 80% agreement with human preferences and flagged position bias, so run each comparison in both answer orders and count a win only when both agree.

Task-specific deterministic checks

Pass-at-k against unit tests for code, schema validation for structured output, math verification for numeric tasks. Where they apply, these checks give the strongest signal at the lowest cost.

The classical metrics persist as fast regression checks. They are no longer the primary signal. Keep them on every build where a reference exists and settings stay fixed, so a sudden drop flags a change worth a closer look.

When to use what in 2026

Common mistakes when using these metrics

• Reporting BLEU on tasks BLEU was not designed for. Asking BLEU to score a chat reply or a creative paragraph misframes the result.
• Using a single reference for tasks with many valid outputs. All three metrics under-score correct alternatives.
• Forgetting tokenization. SacreBLEU exists because raw BLEU comparisons across tokenizers are not reliable.
• Trusting BERTScore on out-of-domain text. The base embedding quality drives the score; a domain-mismatched embedder produces noisy scores.
• Treating any of the three as factuality metrics. None of them are.
• Using only one of the three. Pairing classical metrics with LLM-judge gives a more stable signal.
• Stopping at offline eval. Production scoring on traces with attached scores catches drift the offline eval misses.

How to use this with Future AGI

Future AGI runs reference-based metrics and LLM judges on one platform. BLEU Score and ROUGE Score ship as built-in evals next to METEOR and chrF, and all of them are deterministic: the same input always returns the same score, with no model call. Future AGI’s pricing page lists heuristic evals such as BLEU and ROUGE as always free and unlimited.

For the factuality gap, LLM-as-Judge evals such as Groundedness and Detect Hallucination return a pass or fail with a plain-language reason. Eval scores attach to a span or a whole trace captured by traceAI, which is built on OpenTelemetry, so a ROUGE score and a groundedness verdict sit on the same trace. The platform has an Apache-2.0 core you can self-host from the future-agi repository.

Catching the wrong-direction answer that all three metrics miss

In the revenue example, candidate C reversed the claim and still scored highest on BLEU, ROUGE-L, and BERTScore. Keep the classical metrics where references exist: WMT-style and CNN/DailyMail benchmark continuity, plus cheap regression checks on every build. Put a groundedness or hallucination judge on the same traces so the reversed number gets caught before it reaches a user.

Score your outputs with BLEU, ROUGE, and a groundedness judge in Future AGI

Sources

• Sellam et al. (2020). BLEURT: Learning Robust Metrics for Text Generation
• WMT translation benchmarks
• Google Research ROUGE package (rouge-score) README, including the optional Porter stemmer