Series
LLM Fundamentals
How large language models work, from tokens and embeddings to attention, the Transformer and training, built around “Attention Is All You Need”. Read them in order, or jump to whatever you're curious about.
How a Large Language Model Works: The Big Picture
What really happens when ChatGPT or Llama answers you: tokens, next-token prediction, the Transformer, and why one 2017 paper changed everything.
How LLMs WorkTokenization: How LLMs Turn Text into Tokens
How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.
How LLMs WorkNeural Networks: Weights, Gradient Descent and Backpropagation
How a neural network learns: neurons as weighted sums, loss, gradient descent worked by hand, and backpropagation explained without calculus.
How LLMs WorkEmbeddings and Logits: How LLMs Turn Tokens into Vectors and Back
The table that turns each token into a vector, the dot products that turn a vector back into scores, and why related words point the same way.
How LLMs WorkAttention Explained: Queries, Keys and Values
How attention lets each token pull in context: queries, keys and values worked by hand, why we divide by √d, the causal mask, and the length-squared cost.
How LLMs WorkMulti-Head Attention and the Transformer Block
How a Transformer block works: many attention heads, the MLP, residual connections and layer norm, with shapes and a full GPT-2 parameter count.
How LLMs WorkPositional Encoding: How Transformers Know Word Order
Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.
How LLMs WorkReading "Attention Is All You Need", the 2017 Transformer Paper
A guided reading of the 2017 Transformer paper: encoder, decoder and cross-attention, how it was trained, what it scored, and which choices survived.
How LLMs WorkEncoders, Decoders and GPT, and How Sampling Works
BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.
How LLMs WorkThe Modern LLM Block: What Changed Since 2017
RMSNorm, SwiGLU, no biases, grouped-query attention and RoPE: GPT-2's block next to Llama 3's and Qwen3's, with a full 8.03B parameter count.
How LLMs WorkMixture of Experts: How Giant Models Use a Fraction of Their Weights
How MoE models work: a router picks a few expert MLPs per token, why total and active parameters differ, and how training keeps experts balanced.
How LLMs WorkPretraining: Data, Scaling Laws and Training at Scale
Where pretraining text comes from and how it's filtered, what scaling laws say about model size versus data, and how a run spreads over 16,000 GPUs.
How LLMs WorkFrom Base Model to Assistant: SFT, RLHF and DPO
How a next-token predictor becomes a helpful assistant: supervised fine-tuning, reward models, RLHF with PPO, and DPO, worked with real numbers.
How LLMs WorkReasoning Models: Chain of Thought, RL and Test-Time Compute
How o1 and DeepSeek-R1 learned to think before answering: chain of thought, RL with checkable rewards, GRPO, and what thinking tokens cost.
How LLMs WorkMultimodal Models: How LLMs See and Hear
How images, video and audio become vectors an LLM can read: ViT patches, CLIP, LLaVA-style projectors, Whisper and audio codec tokens, with real numbers.
How LLMs WorkInside the Black Box: Features, Superposition and Circuits
How researchers read an LLM's internals: the logit lens, superposition, sparse autoencoders, circuits and steering, with real numbers and honest limits.
How LLMs Work