Tagged: Tokenization

Articles

• Fundamentals · Part 2How LLMs Work

  Tokenization: How LLMs Turn Text into Tokens

  How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.

• From Scratch · Part 2Training LLMs

  Training a BPE Tokenizer: 40 Lines of Python to 32K Tokens

  Build byte-level BPE from scratch, train the series' 32,768-token tokenizer on FineWeb-Edu, and measure what vocabulary size and training data change.