Tagged: Long context

Articles

• Fundamentals · Part 5How LLMs Work

  Attention Explained: Queries, Keys and Values

  How attention lets each token pull in context: queries, keys and values worked by hand, why we divide by √d, the causal mask, and the length-squared cost.

• Fundamentals · Part 7How LLMs Work

  Positional Encoding: How Transformers Know Word Order

  Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.

• Math · Part 11Math for LLMs

  The Math of Attention: Scaling, Masks, Jacobians and Low Rank

  Attention derived by hand: why divide by √d, the mask as minus infinity, the softmax Jacobian, n² cost, GQA and MLA as factorizations, linear attention.

• Math · Part 12Math for LLMs

  Rotations, Complex Numbers and the Math of RoPE

  Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.

• Inference · Part 7Inference & Optimization

  FlashAttention and Fused Kernels: Attention Without the n×n Matrix

  Why standard attention is memory-bound, how tiling and online softmax compute it exactly in on-chip SRAM, FlashAttention 1 to 4, and fused kernels.

• Inference · Part 11Inference & Optimization

  Long-Context Inference: Sliding Windows, Sparse Attention, KV Eviction

  What 128K to 1M-token prompts cost on a GPU, and how sliding windows, sparse attention, KV cache eviction and hybrids cut attention and memory.

• GPU · Part 10GPUs & Kernels

  Writing an Attention Kernel: FlashAttention and Decode in Triton

  Build a causal FlashAttention kernel and a split-KV, paged decode kernel in Triton, then time them against naive attention and PyTorch's SDPA.