8 articles.
Evals · Part 1Evaluation & Reliability
What a benchmark score claims and what it can't: validity, reliability, saturation and Goodhart's law, with measured probes on ARC, MMLU and MMLU-Pro.
Evals · Part 2Evaluation & Reliability
Who writes benchmark questions, who checks them, and why models now pick which survive. Plus an audit of 5,700 re-checked MMLU questions on Claude.
Evals · Part 3Evaluation & Reliability
A benchmark score is an estimate. 1,140 MMLU questions asked of Claude five times: error bars, clustered errors, paired tests, power and many comparisons.
Evals · Part 4Evaluation & Reliability
Format, few-shot examples, option order, scoring method and answer extraction all move MMLU scores. Measured on four open models, Haiku 5.5 and Sonnet 5.5.
Evals · Part 6Evaluation & Reliability
ROUGE, BERTScore, rubric graders and LLM judges, tested against expert labels: 1,600 summaries and 1,689 pairwise votes, graded by Claude Haiku 5.5.
Evals · Part 7Evaluation & Reliability
Online Elo vs Bradley-Terry on 135,632 real arena votes: order effects, votes needed per gap, style control, and a mini arena judged by Claude.
Evals · Part 8Evaluation & Reliability
pass@k, weak tests and answer checking on HumanEval, HumanEval+ and MATH-500, measured with Claude Haiku 5.5, Sonnet 5.5 and two small Qwen models.
Evals · Part 11Evaluation & Reliability
Decode the conditions behind reported benchmark numbers, then build a 195-question eval of your own: items, power, harness, contamination and grader.
