Math for LLMs

Articles

• Math · Part 1

  The Math You Need for LLMs, and the Shapes Inside One

  Scalars, vectors, matrices and tensors, matrix multiplication as many dot products, and the shape of every tensor in one forward pass of Llama 3.1 8B.

• Math · Part 2

  Vectors and Dot Products: Length, Angle and Meaning

  Length, angle, cosine and projection built from Pythagoras, why random vectors in 4,096 dimensions are nearly perpendicular, and real embedding arithmetic.

• Math · Part 3

  Matrices as Transformations, and Why Neural Networks Need a Bend

  What a matrix does to space, why stacked linear layers collapse into one, how ReLU and SiLU fix that, and how batching changes a matmul's cost.

• Math · Part 4

  Probability and the Chain Rule of Language

  How an LLM scores any text as a product of next-token probabilities, why it adds logs instead, and what sampling, temperature and top-p do to the numbers.

• Math · Part 5

  Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely

  Why softmax uses the exponential, how one subtraction stops it overflowing, what log-sum-exp measures, and the online softmax inside FlashAttention.

• Math · Part 6

  Entropy, Cross-Entropy and Perplexity

  What an LLM's loss number means: surprise, entropy, cross-entropy, perplexity, bits per byte and KL divergence, and why predicting text is compressing it.

• Math · Part 7

  Derivatives, Gradients and the Chain Rule

  The derivative built from the slope of a tiny secant, gradients as the uphill direction, the chain rule and Jacobians, checked with NumPy and PyTorch.

• Math · Part 8

  Backpropagation by Hand: From the Chain Rule to Autograd

  The chain rule as an algorithm: computational graphs, backward passes for matmul, ReLU, GELU and softmax, activation memory, and a 40-line autograd.

• Math · Part 9

  Optimizers: From Gradient Descent to AdamW

  Why gradient descent zig-zags, and how momentum, Adam's bias correction, AdamW's weight decay, warm-up, cosine decay, clipping and Muon fix it.

• Math · Part 10

  Keeping Signals Stable: Initialization, Normalization and Residuals

  Why signals explode or vanish in deep networks, and the math of the fixes: Xavier and He init, LayerNorm and RMSNorm, residuals, pre-norm and μP.

• Math · Part 11

  The Math of Attention: Scaling, Masks, Jacobians and Low Rank

  Attention derived by hand: why divide by √d, the mask as minus infinity, the softmax Jacobian, n² cost, GQA and MLA as factorizations, linear attention.

• Math · Part 12

  Rotations, Complex Numbers and the Math of RoPE

  Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.

• Math · Part 13

  Low Rank and the SVD: How Much of a Matrix Really Matters

  Rank-1 pieces, the SVD as rotate-stretch-rotate, Eckart-Young's best low-rank copy, real Qwen weight spectra, and why LoRA and MLA use thin matrices.

• Math · Part 14

  Floating Point and Quantization: The Math of Fewer Bits

  How computers store decimals, why 256 + 1 = 256 in BF16, why training keeps FP32 copies, and the error math of scales, groups, GPTQ and AWQ on real weights.

• Math · Part 15

  Counting: Parameters, FLOPs, Memory and Scaling Laws

  Exact parameter counts from a config file, 6ND derived with what it leaves out, KV cache bytes, log-log fits, the Chinchilla optimum and MoE routing math.

• Math · Part 16

  The Math of Alignment: Policy Gradients, PPO, DPO and GRPO

  The equations behind RLHF and reasoning RL: the log-derivative trick, baselines, PPO's clip, Bradley-Terry, the KL-regularized optimum, DPO and GRPO.