Tagged: Attention

Articles

• Fundamentals · Part 1How LLMs Work

  How a Large Language Model Works: The Big Picture

  What really happens when ChatGPT or Llama answers you: tokens, next-token prediction, the Transformer, and why one 2017 paper changed everything.

• Fundamentals · Part 5How LLMs Work

  Attention Explained: Queries, Keys and Values

  How attention lets each token pull in context: queries, keys and values worked by hand, why we divide by √d, the causal mask, and the length-squared cost.

• Fundamentals · Part 6How LLMs Work

  Multi-Head Attention and the Transformer Block

  How a Transformer block works: many attention heads, the MLP, residual connections and layer norm, with shapes and a full GPT-2 parameter count.

• Fundamentals · Part 7How LLMs Work

  Positional Encoding: How Transformers Know Word Order

  Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.

• Fundamentals · Part 8How LLMs Work

  Reading "Attention Is All You Need", the 2017 Transformer Paper

  A guided reading of the 2017 Transformer paper: encoder, decoder and cross-attention, how it was trained, what it scored, and which choices survived.

• Fundamentals · Part 9How LLMs Work

  Encoders, Decoders and GPT, and How Sampling Works

  BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.

• Fundamentals · Part 10How LLMs Work

  The Modern LLM Block: What Changed Since 2017

  RMSNorm, SwiGLU, no biases, grouped-query attention and RoPE: GPT-2's block next to Llama 3's and Qwen3's, with a full 8.03B parameter count.

• Math · Part 5Math for LLMs

  Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely

  Why softmax uses the exponential, how one subtraction stops it overflowing, what log-sum-exp measures, and the online softmax inside FlashAttention.

• Math · Part 11Math for LLMs

  The Math of Attention: Scaling, Masks, Jacobians and Low Rank

  Attention derived by hand: why divide by √d, the mask as minus infinity, the softmax Jacobian, n² cost, GQA and MLA as factorizations, linear attention.

• Math · Part 12Math for LLMs

  Rotations, Complex Numbers and the Math of RoPE

  Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.

• From Scratch · Part 4Training LLMs

  Build an LLM From Scratch: Writing the Model

  The series' 88M-parameter Llama-style model in 179 lines of PyTorch, checked against the formula, Hugging Face's Llama, a first loss and an overfit test.

• Interpretability · Part 4Interpretability

  Attention Heads and Induction Heads, Measured in GPT-2

  Find GPT-2 small's previous-token and induction heads, test how they compose, and measure what knocking them out does to in-context learning.

• Interpretability · Part 10Interpretability

  What Training Builds Inside a Model, Snapshot by Snapshot

  Word counts, then word pairs, then induction heads in 300 steps: 49 snapshots of the From Scratch model show what forms when, and what set the timing.

• Inference · Part 7Inference & Optimization

  FlashAttention and Fused Kernels: Attention Without the n×n Matrix

  Why standard attention is memory-bound, how tiling and online softmax compute it exactly in on-chip SRAM, FlashAttention 1 to 4, and fused kernels.

• Inference · Part 11Inference & Optimization

  Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction

  What 128K to 1M-token prompts cost on a GPU, and how sliding windows, sparse attention, KV cache eviction and hybrids cut attention and memory.

• GPU · Part 10GPUs & Kernels

  Writing an Attention Kernel: FlashAttention and Decode in Triton

  Build a causal FlashAttention kernel and a split-KV, paged decode kernel in Triton, then time them against naive attention and PyTorch's SDPA.

• GPU · Part 15GPUs & Kernels

  One Token, Kernel by Kernel: A Decode Step on a Laptop GPU

  Every kernel in one Qwen2.5-1.5B decode step on a laptop RTX 4060: what it does, its time, its place on the roofline, and where the last 15% goes.