Tagged: Rlhf

Articles

• Fundamentals · Part 13How LLMs Work

  From Base Model to Assistant: SFT, RLHF and DPO

  How a next-token predictor becomes a helpful assistant: supervised fine-tuning, reward models, RLHF with PPO, and DPO, worked with real numbers.

• Fundamentals · Part 14How LLMs Work

  Reasoning Models: Chain of Thought, RL and Test-Time Compute

  How o1 and DeepSeek-R1 learned to think before answering: chain of thought, RL with checkable rewards, GRPO, and what thinking tokens cost.

• Math · Part 16Math for LLMs

  The Math of Alignment: Policy Gradients, PPO, DPO and GRPO

  The equations behind RLHF and reasoning RL: the log-derivative trick, baselines, PPO's clip, Bradley-Terry, the KL-regularized optimum, DPO and GRPO.

• From Scratch · Part 11Training LLMs

  Build an LLM From Scratch: Preference Tuning with DPO

  DPO on the 88M chat model: UltraFeedback pairs made its replies longer; its own replies, ranked by a reward model, taught it to stop. Same loss.

• From Scratch · Part 12Training LLMs

  Build an LLM From Scratch: RL with Verifiable Rewards (GRPO)

  GRPO on the 88M model: a checker for addition, 8 replies per problem, reward curves, pass@k before and after, and what RL couldn't teach it.

• Agents · Part 14Agents & Tool Use

  Training Models to Be Agents: Trajectories, Environments, Rewards

  How models learn to act: fine-tuning on agent trajectories, sandboxes by the million, rewards a program can check, multi-turn RL and reward hacking.