NLP Evaluation & Reliability • Sensei Learning System
BLEU, ROUGE, and Perplexity: What Each Metric Actually Measures
Three numbers, three different questions. None of them measures truth.
N-gram overlap — candidate vs reference
Cand
Ref
cyan = shared n-grams (counted) · dark = counted on one side only
Core Concept
These three metrics each answer a different question about a text, and they are
not interchangeable. BLEU asks “how much of the candidate was in the
reference?” (n-gram precision plus a brevity penalty). ROUGE asks “how
much of the reference did the candidate cover?” (recall-oriented overlap, with
ROUGE-L tracking order via the longest common subsequence). Perplexity asks
“how surprised was the language model?” (the exponentiated average
negative log-likelihood of the observed tokens — lower means the sequence was less
surprising to that model).
Key Components
Metric 1
BLEU
n-gram precision against references + a brevity penalty. Built for corpus-level machine-translation comparison.
Metric 2
ROUGE family
Recall-oriented overlap for summarization. ROUGE-N counts shared n-grams; ROUGE-L uses the longest common subsequence, so order matters.
Metric 3
Perplexity
exp(average negative log-likelihood) the model assigns to observed tokens. Lower = less surprising.
The Boundary
What they don’t measure
Lexical overlap ≠ factuality or semantic equivalence. Raw PPL is not comparable across tokenizers, vocabularies, or corpora.
How It Works
- Produce a candidate: a translation, a summary, or a model’s generated sequence.
- BLEU: count candidate n-grams that appear in the reference(s) — precision — then multiply by a brevity penalty so a short string can’t score 100% precision for free. Aggregate across the corpus, not per sentence.
- ROUGE: flip the direction — what fraction of the reference’s n-grams (ROUGE-N) or longest common subsequence (ROUGE-L) does the candidate cover. That recall orientation is why it suits summarization, where you want no key content dropped.
- Perplexity: the model scores each token; average the negative log-likelihood over the sequence and exponentiate it. Read it as an average “branching factor” of surprise.
- Interpret inside the boundary: each number answers its own question. Don’t use one where another was asked, and never compare perplexity across different tokenizers.
Real-World Applications
- Machine-translation benchmarking — WMT-style system comparison with corpus-level BLEU.
- Summarization evaluation — ROUGE against reference summaries (classic shared-task setup).
- Comparing LLM checkpoints on a fixed validation corpus, same tokenizer, before/after a change.
- Drift monitoring — watch validation perplexity over time as a cheap regression alarm for model or data changes.
Checkpoint — answer before you move on
- Why does BLEU score precision against the references, and why is the brevity penalty necessary?
Think about what a two-word candidate that happens to match would score without it.
- Take one faithful paraphrase with low word overlap, and one fluent but factually wrong answer with high overlap. Which one can BLEU or ROUGE rank higher — and what does that expose about what these metrics actually measure?
This is the falsification: overlap is a proxy for agreement, not for truth.
- You score the same text with two different tokenizers and get two different perplexities. Why isn’t the raw PPL comparable — and what would you have to hold fixed to make the comparison meaningful?
Average log-likelihood is per token. If the tokenization changes, the denominator changes.
Guarantee boundary: a high overlap score never certifies correctness, and a perplexity number is only comparable when tokenizer, vocabulary, and corpus are all held constant.
_hero image unavailable this run (image backend outage) — text card delivered._