Tagged: Alignment

Articles

• Fundamentals · Part 13How LLMs Work

  From Base Model to Assistant: SFT, RLHF and DPO

  How a next-token predictor becomes a helpful assistant: supervised fine-tuning, reward models, RLHF with PPO, and DPO, worked with real numbers.

• Fundamentals · Part 16How LLMs Work

  Inside the Black Box: Features, Superposition and Circuits

  How researchers read an LLM's internals: the logit lens, superposition, sparse autoencoders, circuits and steering, with real numbers and honest limits.

• Math · Part 16Math for LLMs

  The Math of Alignment: Policy Gradients, PPO, DPO and GRPO

  The equations behind RLHF and reasoning RL: the log-derivative trick, baselines, PPO's clip, Bradley-Terry, the KL-regularized optimum, DPO and GRPO.

• Interpretability · Part 11Interpretability

  Interpretability for Safety: Lie Detectors, Sycophancy and Audits

  What looking inside a model can and can't check about its honesty: lie and sycophancy probes, model diffing and real audits, measured on Gemma 2 2B.