Tagged: Transformers

Articles

• Fundamentals · Part 1How LLMs Work

  How a Large Language Model Works: The Big Picture

  What really happens when ChatGPT or Llama answers you: tokens, next-token prediction, the Transformer, and why one 2017 paper changed everything.

• Fundamentals · Part 2How LLMs Work

  Tokenization: How LLMs Turn Text into Tokens

  How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.

• Fundamentals · Part 4How LLMs Work

  Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back

  The table that turns each token into a vector, the dot products that turn a vector back into scores, and why related words point the same way.

• Fundamentals · Part 5How LLMs Work

  Attention Explained: Queries, Keys and Values

  How attention lets each token pull in context: queries, keys and values worked by hand, why we divide by √d, the causal mask, and the length-squared cost.

• Fundamentals · Part 6How LLMs Work

  Multi-Head Attention and the Transformer Block

  How a Transformer block works: many attention heads, the MLP, residual connections and layer norm, with shapes and a full GPT-2 parameter count.

• Fundamentals · Part 7How LLMs Work

  Positional Encoding: How Transformers Know Word Order

  Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.

• Fundamentals · Part 8How LLMs Work

  Reading "Attention Is All You Need", the 2017 Transformer Paper

  A guided reading of the 2017 Transformer paper: encoder, decoder and cross-attention, how it was trained, what it scored, and which choices survived.

• Fundamentals · Part 9How LLMs Work

  Encoders, Decoders and GPT, and How Sampling Works

  BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.

• Fundamentals · Part 10How LLMs Work

  The Modern LLM Block: What Changed Since 2017

  RMSNorm, SwiGLU, no biases, grouped-query attention and RoPE: GPT-2's block next to Llama 3's and Qwen3's, with a full 8.03B parameter count.

• Fundamentals · Part 11How LLMs Work

  Mixture of Experts: How Giant Models Use a Fraction of Their Weights

  How MoE models work: a router picks a few expert MLPs per token, why total and active parameters differ, and how training keeps experts balanced.

• Fundamentals · Part 15How LLMs Work

  Multimodal Models: How LLMs See and Hear

  How images, video and audio become vectors an LLM can read: ViT patches, CLIP, LLaVA-style projectors, Whisper and audio codec tokens, with real numbers.

• Fundamentals · Part 16How LLMs Work

  Inside the Black Box: Features, Superposition and Circuits

  How researchers read an LLM's internals: the logit lens, superposition, sparse autoencoders, circuits and steering, with real numbers and honest limits.

• Math · Part 1Math for LLMs

  The Math You Need for LLMs, and the Shapes Inside One

  Scalars, vectors, matrices and tensors, matrix multiplication as many dot products, and the shape of every tensor in one forward pass of Llama 3.1 8B.

• Math · Part 2Math for LLMs

  Vectors and Dot Products: Length, Angle and Meaning

  Length, angle, cosine and projection built from Pythagoras, why random vectors in 4,096 dimensions are nearly perpendicular, and real embedding arithmetic.

• Math · Part 3Math for LLMs

  Matrices as Transformations, and Why Neural Networks Need a Bend

  What a matrix does to space, why stacked linear layers collapse into one, how ReLU and SiLU fix that, and how batching changes a matmul's cost.

• Math · Part 4Math for LLMs

  Probability and the Chain Rule of Language

  How an LLM scores any text as a product of next-token probabilities, why it adds logs instead, and what sampling, temperature and top-p do to the numbers.

• Math · Part 10Math for LLMs

  Keeping Signals Stable: Initialization, Normalization and Residuals

  Why signals explode or vanish in deep networks, and the math of the fixes: Xavier and He init, LayerNorm and RMSNorm, residuals, pre-norm and μP.

• Math · Part 11Math for LLMs

  The Math of Attention: Scaling, Masks, Jacobians and Low Rank

  Attention derived by hand: why divide by √d, the mask as minus infinity, the softmax Jacobian, n² cost, GQA and MLA as factorizations, linear attention.

• Math · Part 12Math for LLMs

  Rotations, Complex Numbers and the Math of RoPE

  Rotation matrices and complex numbers from scratch, a proof that RoPE scores depend only on distance, and the math of PI, NTK, YaRN and ALiBi.

• Math · Part 13Math for LLMs

  Low Rank and the SVD: How Much of a Matrix Really Matters

  Rank-1 pieces, the SVD as rotate-stretch-rotate, Eckart-Young's best low-rank copy, real Qwen weight spectra, and why LoRA and MLA use thin matrices.

• Interpretability · Part 1Interpretability

  Opening the Black Box: What Interpretability Can Explain

  What interpretability is for, its main methods, what counts as an explanation, and the toolkit, tested by asking two methods where GPT-2 keeps facts.

• Interpretability · Part 2Interpretability

  The Residual Stream and the Logit Lens

  Decode a model's half-finished predictions layer by layer with the logit lens, train a tuned lens on a laptop, and see where the simple version breaks.

• Interpretability · Part 3Interpretability

  Linear Probes: Reading What a Model's Activations Encode

  Train one-matrix classifiers on GPT-2 small and Qwen2.5-0.5B to read part of speech and truth from every layer, then check whether the probe did the work.

• Interpretability · Part 4Interpretability

  Attention Heads and Induction Heads, Measured in GPT-2

  Find GPT-2 small's previous-token and induction heads, test how they compose, and measure what knocking them out does to in-context learning.

• Interpretability · Part 5Interpretability

  Activation Patching and Causal Tracing, Measured in GPT-2

  Swap one activation between two prompts to find where GPT-2 tracks names and recalls facts, and measure how corruption and metric choices change the answer.

• Interpretability · Part 6Interpretability

  Superposition: How Models Store More Features Than Neurons

  Reproduce Toy Models of Superposition on a laptop: phase diagrams, pentagons and sticky fractions, then measure how mixed GPT-2 small's MLP neurons are.

• Interpretability · Part 7Interpretability

  Sparse Autoencoders: Pulling Features Out of a Model

  Train a sparse autoencoder on GPT-2 small on a laptop, compare it with public ones, then explore and score Gemma Scope features on Gemma 2 2B.

• Interpretability · Part 8Interpretability

  Circuits and Attribution Graphs, Traced on a Laptop GPU

  From hand-found circuits to attribution patching, transcoders and attribution graphs, run with circuit-tracer on Gemma 2 2B and tested by intervention.