Tagged: Gradient descent

Articles

• Fundamentals · Part 3How LLMs Work

  Neural Networks: Weights, Gradient Descent and Backpropagation

  How a neural network learns: neurons as weighted sums, loss, gradient descent worked by hand, and backpropagation explained without calculus.

• Math · Part 7Math for LLMs

  Derivatives, Gradients and the Chain Rule

  The derivative built from the slope of a tiny secant, gradients as the uphill direction, the chain rule and Jacobians, checked with NumPy and PyTorch.

• Math · Part 8Math for LLMs

  Backpropagation by Hand: From the Chain Rule to Autograd

  The chain rule as an algorithm: computational graphs, backward passes for matmul, ReLU, GELU and softmax, activation memory, and a 40-line autograd.

• Math · Part 9Math for LLMs

  Optimizers: From Gradient Descent to AdamW

  Why gradient descent zig-zags, and how momentum, Adam's bias correction, AdamW's weight decay, warm-up, cosine decay, clipping and Muon fix it.