Series
Measuring LLMs: Evals and Benchmarks
What a benchmark score really tells you, measured: how benchmarks are built, error bars, prompt sensitivity, contamination, LLM judges, arenas, code, math, long-context and safety evals. Read them in order, or jump to whatever you're curious about.
Why Measuring LLMs Is Hard: Validity, Saturation and Goodhart
What a benchmark score claims and what it can't: validity, reliability, saturation and Goodhart's law, with measured probes on ARC, MMLU and MMLU-Pro.
Evaluation & ReliabilityHow Benchmarks Are Built: From MMLU to Humanity's Last Exam
Who writes benchmark questions, who checks them, and why models now pick which survive. Plus an audit of 5,700 re-checked MMLU questions on Claude.
Evaluation & ReliabilityThe Statistics of Evals: Error Bars, Paired Tests and Power
A benchmark score is an estimate. 1,140 MMLU questions asked of Claude five times: error bars, clustered errors, paired tests, power and many comparisons.
Evaluation & ReliabilityPrompt and Harness Sensitivity: Same Model, Same Items, New Score
Format, few-shot examples, option order, scoring method and answer extraction all move MMLU scores. Measured on four open models, Haiku 5.5 and Sonnet 5.5.
Evaluation & ReliabilityBenchmark Contamination: Detecting a Model That Has Seen the Test
We leaked ARC questions into a small model's training data on purpose, then tested n-gram checks, Min-K%, shuffled-order tests and black-box probes on it.
Evaluation & ReliabilityGrading Open-Ended Outputs: Metrics, Rubrics and LLM Judges
ROUGE, BERTScore, rubric graders and LLM judges, tested against expert labels: 1,600 summaries and 1,689 pairwise votes, graded by Claude Haiku 5.5.
Evaluation & ReliabilityArenas and Elo Rankings: How Pairwise Votes Become a Leaderboard
Online Elo vs Bradley-Terry on 135,632 real arena votes: order effects, votes needed per gap, style control, and a mini arena judged by Claude.
Evaluation & ReliabilityCode and Math Benchmarks: What Unit Tests and Answer Checkers Miss
pass@k, weak tests and answer checking on HumanEval, HumanEval+ and MATH-500, measured with Claude Haiku 5.5, Sonnet 5.5 and two small Qwen models.
Evaluation & ReliabilityLong-Context Evals: Needles, RULER and What a Million Tokens Buys
Why needle tests saturate, how RULER and NoLiMa probe the real context size, and Claude Haiku 5.5 and two Qwen models measured from 500 to 505K tokens.
Evaluation & ReliabilitySafety and Capability Evals: Refusals, Red Teams and Sandbagging
How labs test for dangerous capabilities and refusals before release, what those tests can't show, and three measurements: over-refusal, graders, sandbagging.
Evaluation & ReliabilityReading a Model Card and Building Your Own Eval
Decode the conditions behind reported benchmark numbers, then build a 195-question eval of your own: items, power, harness, contamination and grader.
Evaluation & Reliability