From Scratch · Part 1
Build an LLM From Scratch: What a Laptop Can Train
Measured training speed on an 8 GB laptop GPU, 6ND budget math, and the model this series builds: 88M parameters, 1.76B tokens, about 17 hours.
Training LLMs
Tokenizers, data pipelines, training loops, scaling laws and post-training: how a language model is built and trained.
Articles
From Scratch · Part 2
Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens
Build byte-level BPE from scratch, train the series' 32,768-token tokenizer on FineWeb-Edu, and measure what vocabulary size and training data change.
From Scratch · Part 3
The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Download FineWeb-Edu, check its filters, remove near-duplicates with MinHash, then tokenize, shuffle and pack the series' training data into token shards.
From Scratch · Part 4
Build an LLM From Scratch: Writing the Model
The series' 88M-parameter Llama-style model in 179 lines of PyTorch, checked against the formula, Hugging Face's Llama, a first loss and an overfit test.
From Scratch · Part 5
Build an LLM From Scratch: The Training Loop
train.py for the series' model: AdamW, warm-up and cosine decay, clipping, BF16, gradient accumulation, exact resume, and a first real loss curve.
From Scratch · Part 6
Build an LLM From Scratch: Making Training Fast on One GPU
From 5,800 to 30,000 tokens per second on a laptop GPU: TF32, BF16, FlashAttention, torch.compile, fused AdamW and batch size, each measured on its own.
From Scratch · Part 7
Build an LLM From Scratch: The Pretraining Run
16.8 hours, 1.76 billion tokens, 0.83 kWh: the 88M model's run on a laptop GPU, read through its loss curves, spikes, held-out loss and samples.
From Scratch · Part 8
Build an LLM From Scratch: Scaling Laws on a Laptop
Five model sizes from 11M to 47M parameters in 12 hours on a laptop GPU, a fitted power law, and a direct test of Chinchilla's 20 tokens per parameter.
From Scratch · Part 9
Build an LLM From Scratch: Evaluating a Base Model
Perplexity, bits per byte and zero-shot multiple choice for the 88M model against GPT-2, Pythia and SmolLM2, with a harness checked against lm-eval.
From Scratch · Part 10
Build an LLM From Scratch: Supervised Fine-Tuning
Turning the 88M base model into a chat model in 15 minutes: a chat template on reserved tokens, masked loss, 26,000 conversations, and what it cost.
From Scratch · Part 11
Build an LLM From Scratch: Preference Tuning with DPO
DPO on the 88M chat model: UltraFeedback pairs made its replies longer; its own replies, ranked by a reward model, taught it to stop. Same loss.
From Scratch · Part 12
Build an LLM From Scratch: RL with Verifiable Rewards (GRPO)
GRPO on the 88M model: a checker for addition, 8 replies per problem, reward curves, pass@k before and after, and what RL couldn't teach it.
From Scratch · Part 13
Build an LLM From Scratch: Distillation and Synthetic Data
A 15M student learns from our 88M model's probabilities, and the 88M model learns from replies an open 1.5B model wrote: costs, filters, results.
From Scratch · Part 14
Build an LLM From Scratch: Fitting Bigger Models in Memory
Where 8 GB goes when training the 88M model, predicted and measured; activation checkpointing, 8-bit AdamW and offload timed; ZeRO and FSDP explained.
From Scratch · Part 15
Build an LLM From Scratch: From Our Model to Frontier Models
Our 88M run next to Llama 3, DeepSeek-V3, OLMo 2 and SmolLM, the model exported to Hugging Face and llama.cpp, and the whole series reviewed.