Every model launch arrives holding a fistful of numbers: an MMLU score here, an Arena rank there, a pass rate on a coding set. Our evals explainer shows you how to build an evaluation for your own workload; this lesson gives you the LLM evaluation terms for reading everyone’s. The LLM concepts deep dive shows where these benchmark families sit in the bigger picture. Half of them name what is being measured, and the other half name why the number on the slide may not mean what the announcement implies.
What an evaluation is made of
Four LLM evaluation terms for the parts of any measurement exercise.
- Evals: Short for evaluation: running a model against a defined workload and scoring the results, usually on your own data and on a schedule. Evals are broader than benchmarks because they can be private, continuous, and tied directly to the product decision you need to make.
- Benchmark: A fixed, public dataset plus a scoring rule, published so results can be compared across models and over time. Being fixed is both the feature and the flaw: comparability today, saturation and contamination tomorrow.
- Held-out set: Data deliberately excluded from training so scores on it estimate how the model handles material it has never seen. If any of it leaks into pretraining, the estimate stops estimating anything, which is the contamination entry further down this page.
- Golden dataset: A small, curated set of questions with reference answers trusted enough to grade against. Teams keep one as the referee inside production evals, updating it as the product’s definition of correct sharpens.
The benchmarks you will keep reading about
Three names that carry most of the weight in model announcements.
- MMLU: Multiple-choice questions spanning academic and professional subjects, for years the default proxy for broad knowledge. Scores now sit close enough to the ceiling that its successor, MMLU-Pro, with harder questions and more decoys, does most of the separating.
- GSM8K and MATH: Grade-school word problems and competition mathematics respectively, historically the place where reasoning differences appeared first. Both are effectively retired as discriminators; their descendants push the difficulty instead.
- Chatbot Arena: Blind pairwise votes from real users, converted to Elo-style ratings. It measures what people prefer, not what is correct, and it keeps working after knowledge benchmarks saturate, which is why launch posts lean on it ever harder.
How outputs get scored
Three terms for the grading layer that turns answers into numbers.
- LLM-as-a-judge: Delegating the grading to a model that scores each answer against a rubric, currently the only way grading scales to thousands of open-ended responses. Its known biases are cataloged: favoring the first option shown, favoring length, and mild self-preference, so serious setups randomize order and calibrate the judge against humans before trusting it.
- Pass@k: The chance that at least one sample out of k attempts passes the checks, the standard metric on coding benchmarks such as HumanEval and SWE-bench. It bakes in a cost tradeoff: k samples means k times the compute, and sampling one strong answer beats sampling ten mediocre ones.
- BLEU and ROUGE: Word-overlap metrics against a reference text, from the translation and summarization eras. They survive because they are free and deterministic, but they reward lexical similarity rather than meaning, which is usually the thing you wanted measured.
Why scores mislead
Two terms that decide whether a leaderboard number deserves your trust.
- Data contamination: Test material that was already inside the training corpus, whether by leakage, licensing, or the open web simply containing the benchmark. The model recognizes the questions instead of answering them, and reported scores drift upward with no matching capability behind them.
- Goodhart’s law: The observation that once a measure becomes a target it stops measuring well. In this field it arrives twice: benchmark questions end up in training mixes by accident, and vendors tune directly for famous benchmarks on purpose. Saturation is the benign form of the same pressure.
Common confusions
Three pairs that get conflated in evaluation discussions.
- Evals vs benchmarks: A benchmark is public, fixed, and meant for comparing models. Evals are yours: private, versioned, and meant for deciding whether to ship. Strong products need both, and only one of them ever shows up in a launch post.
- Contamination vs saturation: Contamination means the test leaked into training and the score is partly fake. Saturation means the model family genuinely handles the material and the test no longer separates contenders. Same flat scoreboard, opposite remedies: replace the data in one case, raise the difficulty in the other.
- Arena rank vs accuracy score: An accuracy benchmark grades against fixed correct answers. The Arena grades model against model through user preference, which folds in style, confidence, and formatting alongside correctness, so a model can climb one while sliding on the other.
Further reading
Three papers worth the time if you run evaluations.
- HELM, the Holistic Evaluation of Language Models, the framework that turned transparency about scenarios and metrics into a checklist you can copy.
- Chatbot Arena, the paper behind preference-based leaderboards, including the methodology that keeps the votes meaningful.
- Judging LLM-as-a-Judge, the study that cataloged the judge biases your grading pipeline needs to defend against.
This is lesson 12 of the AI glossary. Lesson 11 sorted the alignment terms in Alignment, RLHF and Reinforcement Learning, and evaluation is exactly how anyone checks that alignment work moved anything at all. For building your own pipeline, LLM Evals Explained covers it end to end. The next lesson turns from measuring outputs to the questions underneath deployment: safety, security, and interpretability.