Tagged: Multilingual

Articles

• Fundamentals · Part 2How LLMs Work

  Tokenization: How LLMs Turn Text into Tokens

  How byte-pair encoding builds a vocabulary, why models stumble on spelling and arithmetic, and why the same sentence costs more tokens in Hindi.