Fundamentals · Part 1How LLMs Work
How a Large Language Model Works: The Big Picture
What really happens when ChatGPT or Llama answers you: tokens, next-token prediction, the Transformer, and why one 2017 paper changed everything.
Tagged: Tokens
9 articles.
Articles
Fundamentals · Part 2How LLMs Work
Tokenization: How LLMs Turn Text into Tokens
How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.
Fundamentals · Part 4How LLMs Work
Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back
The table that turns each token into a vector, the dot products that turn a vector back into scores, and why related words point the same way.
Fundamentals · Part 15How LLMs Work
Multimodal Models: How LLMs See and Hear
How images, video and audio become vectors an LLM can read: ViT patches, CLIP, LLaVA-style projectors, Whisper and audio codec tokens, with real numbers.
Math · Part 1Math for LLMs
The Math You Need for LLMs, and the Shapes Inside One
Scalars, vectors, matrices and tensors, matrix multiplication as many dot products, and the shape of every tensor in one forward pass of Llama 3.1 8B.
Math · Part 4Math for LLMs
Probability and the Chain Rule of Language
How an LLM scores any text as a product of next-token probabilities, why it adds logs instead, and what sampling, temperature and top-p do to the numbers.
Math · Part 6Math for LLMs
Entropy, Cross-Entropy and Perplexity
What an LLM's loss number means: surprise, entropy, cross-entropy, perplexity, bits per byte and KL divergence, and why predicting text is compressing it.
From Scratch · Part 2Training LLMs
Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens
Build byte-level BPE from scratch, train the series' 32,768-token tokenizer on FineWeb-Edu, and measure what vocabulary size and training data change.
From Scratch · Part 3Training LLMs
The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Download FineWeb-Edu, check its filters, remove near-duplicates with MinHash, then tokenize, shuffle and pack the series' training data into token shards.