Fundamentals · Part 12How LLMs Work
Pretraining: Data, Scaling Laws and Training at Scale
Where pretraining text comes from and how it's filtered, what scaling laws say about model size versus data, and how a run spreads over 16,000 GPUs.
Tagged: Data
1 article.