From Scratch · Part 1Training LLMs
Build an LLM From Scratch: What a Laptop Can Train
Measured training speed on an 8 GB laptop GPU, 6ND budget math, and the model this series builds: 88M parameters, 1.76B tokens, about 17 hours.
Tagged: Pretraining
10 articles.
Articles
From Scratch · Part 2Training LLMs
Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens
Build byte-level BPE from scratch, train the series' 32,768-token tokenizer on FineWeb-Edu, and measure what vocabulary size and training data change.
From Scratch · Part 3Training LLMs
The Pretraining Data Pipeline: 2.5 Billion Tokens on a Laptop
Download FineWeb-Edu, check its filters, remove near-duplicates with MinHash, then tokenize, shuffle and pack the series' training data into token shards.
From Scratch · Part 4Training LLMs
Build an LLM From Scratch: Writing the Model
The series' 88M-parameter Llama-style model in 179 lines of PyTorch, checked against the formula, Hugging Face's Llama, a first loss and an overfit test.
From Scratch · Part 5Training LLMs
Build an LLM From Scratch: The Training Loop
train.py for the series' model: AdamW, warm-up and cosine decay, clipping, BF16, gradient accumulation, exact resume, and a first real loss curve.
From Scratch · Part 6Training LLMs
Build an LLM From Scratch: Making Training Fast on One GPU
From 5,800 to 30,000 tokens per second on a laptop GPU: TF32, BF16, FlashAttention, torch.compile, fused AdamW and batch size, each measured on its own.
From Scratch · Part 7Training LLMs
Build an LLM From Scratch: The Pretraining Run
16.8 hours, 1.76 billion tokens, 0.83 kWh: the 88M model's run on a laptop GPU, read through its loss curves, spikes, held-out loss and samples.
From Scratch · Part 8Training LLMs
Build an LLM From Scratch: Scaling Laws on a Laptop
Five model sizes from 11M to 47M parameters in 12 hours on a laptop GPU, a fitted power law, and a direct test of Chinchilla's 20 tokens per parameter.
From Scratch · Part 9Training LLMs
Build an LLM From Scratch: Evaluating a Base Model
Perplexity, bits per byte and zero-shot multiple choice for the 88M model against GPT-2, Pythia and SmolLM2, with a harness checked against lm-eval.
From Scratch · Part 15Training LLMs
Build an LLM From Scratch: From Our Model to Frontier Models
Our 88M run next to Llama 3, DeepSeek-V3, OLMo 2 and SmolLM, the model exported to Hugging Face and llama.cpp, and the whole series reviewed.