Tagged: Benchmarks

Articles

• Evals · Part 1Evaluation & Reliability

  Why Measuring LLMs Is Hard: Validity, Saturation and Goodhart

  What a benchmark score claims and what it can't: validity, reliability, saturation and Goodhart's law, with measured probes on ARC, MMLU and MMLU-Pro.

• Evals · Part 2Evaluation & Reliability

  How Benchmarks Are Built: From MMLU to Humanity's Last Exam

  Who writes benchmark questions, who checks them, and why models now pick which survive. Plus an audit of 5,700 re-checked MMLU questions on Claude.

• Evals · Part 3Evaluation & Reliability

  The Statistics of Evals: Error Bars, Paired Tests and Power

  A benchmark score is an estimate. 1,140 MMLU questions asked of Claude five times: error bars, clustered errors, paired tests, power and many comparisons.

• Evals · Part 4Evaluation & Reliability

  Prompt and Harness Sensitivity: Same Model, Same Items, New Score

  Format, few-shot examples, option order, scoring method and answer extraction all move MMLU scores. Measured on four open models, Haiku 5.5 and Sonnet 5.5.

• Evals · Part 5Evaluation & Reliability

  Benchmark Contamination: Detecting a Model That Has Seen the Test

  We leaked ARC questions into a small model's training data on purpose, then tested n-gram checks, Min-K%, shuffled-order tests and black-box probes on it.

• Evals · Part 6Evaluation & Reliability

  Grading Open-Ended Outputs: Metrics, Rubrics and LLM Judges

  ROUGE, BERTScore, rubric graders and LLM judges, tested against expert labels: 1,600 summaries and 1,689 pairwise votes, graded by Claude Haiku 5.5.

• Evals · Part 7Evaluation & Reliability

  Arenas and Elo Rankings: How Pairwise Votes Become a Leaderboard

  Online Elo vs Bradley-Terry on 135,632 real arena votes: order effects, votes needed per gap, style control, and a mini arena judged by Claude.

• Evals · Part 8Evaluation & Reliability

  Code and Math Benchmarks: What Unit Tests and Answer Checkers Miss

  pass@k, weak tests and answer checking on HumanEval, HumanEval+ and MATH-500, measured with Claude Haiku 5.5, Sonnet 5.5 and two small Qwen models.

• Evals · Part 9Evaluation & Reliability

  Long-Context Evals: Needles, RULER and What a Million Tokens Buys

  Why needle tests saturate, how RULER and NoLiMa probe the real context size, and Claude Haiku 5.5 and two Qwen models measured from 500 to 505K tokens.

• Evals · Part 10Evaluation & Reliability

  Safety and Capability Evals: Refusals, Red Teams and Sandbagging

  How labs test for dangerous capabilities and refusals before release, what those tests can't show, and three measurements: over-refusal, graders, sandbagging.

• Evals · Part 11Evaluation & Reliability

  Reading a Model Card and Building Your Own Eval

  Decode the conditions behind reported benchmark numbers, then build a 195-question eval of your own: items, power, harness, contamination and grader.