Series

Inside the Model: Interpretability

Look inside real language models on one laptop GPU: the logit lens, probes, induction heads, activation patching, superposition, sparse autoencoders, circuits and steering, measured. Read them in order, or jump to whatever you're curious about.

  1. Opening the Black Box: What Interpretability Can Explain

    What interpretability is for, its main methods, what counts as an explanation, and the toolkit, tested by asking two methods where GPT-2 keeps facts.

    Interpretability

  2. The Residual Stream and the Logit Lens

    Decode a model's half-finished predictions layer by layer with the logit lens, train a tuned lens on a laptop, and see where the simple version breaks.

    Interpretability

  3. Linear Probes: Reading What a Model's Activations Encode

    Train one-matrix classifiers on GPT-2 small and Qwen2.5-0.5B to read part of speech and truth from every layer, then check whether the probe did the work.

    Interpretability

  4. Attention Heads and Induction Heads, Measured in GPT-2

    Find GPT-2 small's previous-token and induction heads, test how they compose, and measure what knocking them out does to in-context learning.

    Interpretability

  5. Activation Patching and Causal Tracing, Measured in GPT-2

    Swap one activation between two prompts to find where GPT-2 tracks names and recalls facts, and measure how corruption and metric choices change the answer.

    Interpretability

  6. Superposition: How Models Store More Features Than Neurons

    Reproduce Toy Models of Superposition on a laptop: phase diagrams, pentagons and sticky fractions, then measure how mixed GPT-2 small's MLP neurons are.

    Interpretability

  7. Sparse Autoencoders: Pulling Features Out of a Model

    Train a sparse autoencoder on GPT-2 small on a laptop, compare it with public ones, then explore and score Gemma Scope features on Gemma 2 2B.

    Interpretability

  8. Circuits and Attribution Graphs, Traced on a Laptop GPU

    From hand-found circuits to attribution patching, transcoders and attribution graphs, run with circuit-tracer on Gemma 2 2B and tested by intervention.

    Interpretability

  9. Steering With Activations: The Refusal Direction, Measured

    Find one direction that controls refusal in an open chat model, remove it and add it, measure the side effects, and compare SAE steering with a mean difference.

    Interpretability

  10. What Training Builds Inside a Model, Snapshot by Snapshot

    Word counts, then word pairs, then induction heads in 300 steps: 49 snapshots of the From Scratch model show what forms when, and what set the timing.

    Interpretability

  11. Interpretability for Safety: Lie Detectors, Sycophancy and Audits

    What looking inside a model can and can't check about its honesty: lie and sycophancy probes, model diffing and real audits, measured on Gemma 2 2B.

    Interpretability

  12. Limits and Open Problems in Interpretability

    The series in review: every part's headline result, a robustness audit of three of them, the field's sharpest critiques, and the problems still open.

    Interpretability