How LLMs Work

Articles

• Fundamentals · Part 1

  How a Large Language Model Works: The Big Picture

  What really happens when ChatGPT or Llama answers you: tokens, next-token prediction, the Transformer, and why one 2017 paper changed everything.

• Fundamentals · Part 2

  Tokenization: How LLMs Turn Text into Tokens

  How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.

• Fundamentals · Part 3

  Neural Networks: Weights, Gradient Descent and Backpropagation

  How a neural network learns: neurons as weighted sums, loss, gradient descent worked by hand, and backpropagation explained without calculus.

• Fundamentals · Part 4

  Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back

  The table that turns each token into a vector, the dot products that turn a vector back into scores, and why related words point the same way.

• Fundamentals · Part 5

  Attention Explained: Queries, Keys and Values

  How attention lets each token pull in context: queries, keys and values worked by hand, why we divide by √d, the causal mask, and the length-squared cost.

• Fundamentals · Part 6

  Multi-Head Attention and the Transformer Block

  How a Transformer block works: many attention heads, the MLP, residual connections and layer norm, with shapes and a full GPT-2 parameter count.

• Fundamentals · Part 7

  Positional Encoding: How Transformers Know Word Order

  Attention can't see word order on its own. How sinusoids, learned tables, RoPE and ALiBi add it, and how models stretch their context past training length.

• Fundamentals · Part 8

  Reading "Attention Is All You Need", the 2017 Transformer Paper

  A guided reading of the 2017 Transformer paper: encoder, decoder and cross-attention, how it was trained, what it scored, and which choices survived.

• Fundamentals · Part 9

  Encoders, Decoders and GPT, and How Sampling Works

  BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.

• Fundamentals · Part 10

  The Modern LLM Block: What Changed Since 2017

  RMSNorm, SwiGLU, no biases, grouped-query attention and RoPE: GPT-2's block next to Llama 3's and Qwen3's, with a full 8.03B parameter count.

• Fundamentals · Part 11

  Mixture of Experts: How Giant Models Use a Fraction of Their Weights

  How MoE models work: a router picks a few expert MLPs per token, why total and active parameters differ, and how training keeps experts balanced.

• Fundamentals · Part 12

  Pretraining: Data, Scaling Laws and Training at Scale

  Where pretraining text comes from and how it's filtered, what scaling laws say about model size versus data, and how a run spreads over 16,000 GPUs.

• Fundamentals · Part 13

  From Base Model to Assistant: SFT, RLHF and DPO

  How a next-token predictor becomes a helpful assistant: supervised fine-tuning, reward models, RLHF with PPO, and DPO, worked with real numbers.

• Fundamentals · Part 14

  Reasoning Models: Chain of Thought, RL and Test-Time Compute

  How o1 and DeepSeek-R1 learned to think before answering: chain of thought, RL with checkable rewards, GRPO, and what thinking tokens cost.

• Fundamentals · Part 15

  Multimodal Models: How LLMs See and Hear

  How images, video and audio become vectors an LLM can read: ViT patches, CLIP, LLaVA-style projectors, Whisper and audio codec tokens, with real numbers.

• Fundamentals · Part 16

  Inside the Black Box: Features, Superposition and Circuits

  How researchers read an LLM's internals: the logit lens, superposition, sparse autoencoders, circuits and steering, with real numbers and honest limits.