Series

Build an LLM From Scratch

Train a small GPT-style model on one laptop GPU, step by step: tokenizer, data pipeline, model, training loop, scaling laws, evaluation, then fine-tuning, DPO and RL. Read them in order, or jump to whatever you're curious about.

  1. Build an LLM From Scratch: What a Laptop Can Train

    Measured training speed on an 8 GB laptop GPU, 6ND budget math, and the model this series builds: 88M parameters, 1.76B tokens, about 17 hours.

    Training LLMs

  2. Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens

    Build byte-level BPE from scratch, train the series' 32,768-token tokenizer on FineWeb-Edu, and measure what vocabulary size and training data change.

    Training LLMs

  3. The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop

    Download FineWeb-Edu, check its filters, remove near-duplicates with MinHash, then tokenize, shuffle and pack the series' training data into token shards.

    Training LLMs

  4. Build an LLM From Scratch: Writing the Model

    The series' 88M-parameter Llama-style model in 179 lines of PyTorch, checked against the formula, Hugging Face's Llama, a first loss and an overfit test.

    Training LLMs

  5. Build an LLM From Scratch: The Training Loop

    train.py for the series' model: AdamW, warm-up and cosine decay, clipping, BF16, gradient accumulation, exact resume, and a first real loss curve.

    Training LLMs

  6. Build an LLM From Scratch: Making Training Fast on One GPU

    From 5,800 to 30,000 tokens per second on a laptop GPU: TF32, BF16, FlashAttention, torch.compile, fused AdamW and batch size, each measured on its own.

    Training LLMs

  7. Build an LLM From Scratch: The Pretraining Run

    16.8 hours, 1.76 billion tokens, 0.83 kWh: the 88M model's run on a laptop GPU, read through its loss curves, spikes, held-out loss and samples.

    Training LLMs

  8. Build an LLM From Scratch: Scaling Laws on a Laptop

    Five model sizes from 11M to 47M parameters in 12 hours on a laptop GPU, a fitted power law, and a direct test of Chinchilla's 20 tokens per parameter.

    Training LLMs

  9. Build an LLM From Scratch: Evaluating a Base Model

    Perplexity, bits per byte and zero-shot multiple choice for the 88M model against GPT-2, Pythia and SmolLM2, with a harness checked against lm-eval.

    Training LLMs

  10. Build an LLM From Scratch: Supervised Fine-Tuning

    Turning the 88M base model into a chat model in 15 minutes: a chat template on reserved tokens, masked loss, 26,000 conversations, and what it cost.

    Training LLMs

  11. Build an LLM From Scratch: Preference Tuning with DPO

    DPO on the 88M chat model: UltraFeedback pairs made its replies longer; its own replies, ranked by a reward model, taught it to stop. Same loss.

    Training LLMs

  12. Build an LLM From Scratch: RL with Verifiable Rewards (GRPO)

    GRPO on the 88M model: a checker for addition, 8 replies per problem, reward curves, pass@k before and after, and what RL couldn't teach it.

    Training LLMs

  13. Build an LLM From Scratch: Distillation and Synthetic Data

    A 15M student learns from our 88M model's probabilities, and the 88M model learns from replies an open 1.5B model wrote: costs, filters, results.

    Training LLMs

  14. Build an LLM From Scratch: Fitting Bigger Models in Memory

    Where 8 GB goes when training the 88M model, predicted and measured; activation checkpointing, 8-bit AdamW and offload timed; ZeRO and FSDP explained.

    Training LLMs

  15. Build an LLM From Scratch: From Our Model to Frontier Models

    Our 88M run next to Llama 3, DeepSeek-V3, OLMo 2 and SmolLM, the model exported to Hugging Face and llama.cpp, and the whole series reviewed.

    Training LLMs