Tagged: Superposition

Articles

• Interpretability · Part 6Interpretability

  Superposition: How Models Store More Features Than Neurons

  Reproduce Toy Models of Superposition on a laptop: phase diagrams, pentagons and sticky fractions, then measure how mixed GPT-2 small's MLP neurons are.

• Interpretability · Part 7Interpretability

  Sparse Autoencoders: Pulling Features Out of a Model

  Train a sparse autoencoder on GPT-2 small on a laptop, compare it with public ones, then explore and score Gemma Scope features on Gemma 2 2B.