Tagged: Optimizers

Articles

• Math · Part 9Math for LLMs

  Optimizers: From Gradient Descent to AdamW

  Why gradient descent zig-zags, and how momentum, Adam's bias correction, AdamW's weight decay, warm-up, cosine decay, clipping and Muon fix it.

• From Scratch · Part 5Training LLMs

  Build an LLM From Scratch: The Training Loop

  train.py for the series' model: AdamW, warm-up and cosine decay, clipping, BF16, gradient accumulation, exact resume, and a first real loss curve.