Tagged: Positional encoding

Articles

• Fundamentals · Part 7How LLMs Work

  Positional Encoding: How Transformers Know Word Order

  Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.

• Math · Part 12Math for LLMs

  Rotations, Complex Numbers and the Math of RoPE

  Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.