Tagged: In context learning

Articles

• Interpretability · Part 4Interpretability

  Attention Heads and Induction Heads, Measured in GPT-2

  Find GPT-2 small's previous-token and induction heads, test how they compose, and measure what knocking them out does to in-context learning.

• Interpretability · Part 10Interpretability

  What Training Builds Inside a Model, Snapshot by Snapshot

  Word counts, then word pairs, then induction heads in 300 steps: 49 snapshots of the From Scratch model show what forms when, and what set the timing.