Tagged: Embeddings

Articles

• Fundamentals · Part 4How LLMs Work

  Embeddings and Logits: How LLMs Turn Tokens into Vectors and Back

  The table that turns each token into a vector, the dot products that turn a vector back into scores, and why related words point the same way.

• Fundamentals · Part 9How LLMs Work

  Encoders, Decoders and GPT, and How Sampling Works

  BERT, GPT and T5 share one Transformer and differ in what each token may see. Why decoder-only won, and how top-k, top-p and min-p pick tokens.

• Fundamentals · Part 15How LLMs Work

  Multimodal Models: How LLMs See and Hear

  How images, video and audio become vectors an LLM can read: ViT patches, CLIP, LLaVA-style projectors, Whisper and audio codec tokens, with real numbers.

• Fundamentals · Part 16How LLMs Work

  Inside the Black Box: Features, Superposition and Circuits

  How researchers read an LLM's internals: the logit lens, superposition, sparse autoencoders, circuits and steering, with real numbers and honest limits.

• Math · Part 1Math for LLMs

  The Math You Need for LLMs, and the Shapes Inside One

  Scalars, vectors, matrices and tensors, matrix multiplication as many dot products, and the shape of every tensor in one forward pass of Llama 3.1 8B.

• Math · Part 2Math for LLMs

  Vectors and Dot Products: Length, Angle and Meaning

  Length, angle, cosine and projection built from Pythagoras, why random vectors in 4,096 dimensions are nearly perpendicular, and real embedding arithmetic.

• Interpretability · Part 3Interpretability

  Linear Probes: Reading What a Model's Activations Encode

  Train one-matrix classifiers on GPT-2 small and Qwen2.5-0.5B to read part of speech and truth from every layer, then check whether the probe did the work.