Series

The Math Behind LLMs

The math inside a language model, built up from high-school algebra: vectors, probability, gradients, backpropagation, attention, RoPE and the equations behind training and alignment. Read them in order, or jump to whatever you're curious about.

  1. The Math You Need for LLMs, and the Shapes Inside One

    Scalars, vectors, matrices and tensors, matrix multiplication as many dot products, and the shape of every tensor in one forward pass of Llama 3.1 8B.

    Math for LLMs

  2. Vectors and Dot Products: Length, Angle and Meaning

    Length, angle, cosine and projection built from Pythagoras, why random vectors in 4,096 dimensions are nearly perpendicular, and real embedding arithmetic.

    Math for LLMs

  3. Matrices as Transformations, and Why Neural Networks Need a Bend

    What a matrix does to space, why stacked linear layers collapse into one, how ReLU and SiLU fix that, and how batching changes a matmul's cost.

    Math for LLMs

  4. Probability and the Chain Rule of Language

    How an LLM scores any text as a product of next-token probabilities, why it adds logs instead, and what sampling, temperature and top-p do to the numbers.

    Math for LLMs

  5. Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely

    Why softmax uses the exponential, how one subtraction stops it overflowing, what log-sum-exp measures, and the online softmax inside FlashAttention.

    Math for LLMs

  6. Entropy, Cross-Entropy and Perplexity

    What an LLM's loss number means: surprise, entropy, cross-entropy, perplexity, bits per byte and KL divergence, and why predicting text is compressing it.

    Math for LLMs

  7. Derivatives, Gradients and the Chain Rule

    The derivative built from the slope of a tiny secant, gradients as the uphill direction, the chain rule and Jacobians, checked with NumPy and PyTorch.

    Math for LLMs

  8. Backpropagation by Hand: From the Chain Rule to Autograd

    The chain rule as an algorithm: computational graphs, backward passes for matmul, ReLU, GELU and softmax, activation memory, and a 40-line autograd.

    Math for LLMs

  9. Optimizers: From Gradient Descent to AdamW

    Why gradient descent zig-zags, and how momentum, Adam's bias correction, AdamW's weight decay, warm-up, cosine decay, clipping and Muon fix it.

    Math for LLMs

  10. Keeping Signals Stable: Initialization, Normalization and Residuals

    Why signals explode or vanish in deep networks, and the math of the fixes: Xavier and He init, LayerNorm and RMSNorm, residuals, pre-norm and μP.

    Math for LLMs

  11. The Math of Attention: Scaling, Masks, Jacobians and Low Rank

    Attention derived by hand: why divide by √d, the mask as minus infinity, the softmax Jacobian, n² cost, GQA and MLA as factorizations, linear attention.

    Math for LLMs

  12. Rotations, Complex Numbers and the Math of RoPE

    Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.

    Math for LLMs

  13. Low Rank and the SVD: How Much of a Matrix Really Matters

    Rank-1 pieces, the SVD as rotate-stretch-rotate, Eckart-Young's best low-rank copy, real Qwen weight spectra, and why LoRA and MLA use thin matrices.

    Math for LLMs

  14. Floating Point and Quantization: The Math of Fewer Bits

    How computers store decimals, why 256 + 1 = 256 in BF16, why training keeps FP32 copies, and the error math of scales, groups, GPTQ and AWQ on real weights.

    Math for LLMs

  15. Counting: Parameters, FLOPs, Memory and Scaling Laws

    Exact parameter counts from a config file, 6ND derived with what it leaves out, KV cache bytes, log-log fits, the Chinchilla optimum and MoE routing math.

    Math for LLMs

  16. The Math of Alignment: Policy Gradients, PPO, DPO and GRPO

    The equations behind RLHF and reasoning RL: the log-derivative trick, baselines, PPO's clip, Bradley-Terry, the KL-regularized optimum, DPO and GRPO.

    Math for LLMs