Tagged: Scaling laws

Articles

• Fundamentals · Part 12How LLMs Work

  Pretraining: Data, Scaling Laws and Training at Scale

  Where pretraining text comes from and how it's filtered, what scaling laws say about model size versus data, and how a run spreads over 16,000 GPUs.

• Math · Part 15Math for LLMs

  Counting: Parameters, FLOPs, Memory and Scaling Laws

  Exact parameter counts from a config file, 6ND derived with what it leaves out, KV cache bytes, log-log fits, the Chinchilla optimum and MoE routing math.

• From Scratch · Part 1Training LLMs

  Build an LLM From Scratch: What a Laptop Can Train

  Measured training speed on an 8 GB laptop GPU, 6ND budget math, and the model this series builds: 88M parameters, 1.76B tokens, about 17 hours.

• From Scratch · Part 8Training LLMs

  Build an LLM From Scratch: Scaling Laws on a Laptop

  Five model sizes from 11M to 47M parameters in 12 hours on a laptop GPU, a fitted power law, and a direct test of Chinchilla's 20 tokens per parameter.