Tagged: Sampling

Articles

• Fundamentals · Part 9How LLMs Work

  Encoders, Decoders and GPT, and How Sampling Works

  BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.

• Math · Part 4Math for LLMs

  Probability and the Chain Rule of Language

  How an LLM scores any text as a product of next-token probabilities, why it adds logs instead, and what sampling, temperature and top-p do to the numbers.

• Math · Part 5Math for LLMs

  Softmax and Log-Sum-Exp: From Scores to Probabilities, Safely

  Why softmax uses the exponential, how one subtraction stops it overflowing, what log-sum-exp measures, and the online softmax inside FlashAttention.