Mechanistic Interpretability
In May 2024, Anthropic published Scaling Monosemanticity and showed that a single internal feature in Claude 3 Sonnet fires for the Golden Gate Bridge across English prose, French prose, photographs of the bridge, and source code that references it. When the researchers clamped that feature's activation high and asked the model "what are you," it answered as if it were the bridge. The field of mechanistic interpretability had spent half a decade arguing whether large language models contained human-readable concepts at all. One paper turned the question into an engineering problem.
How it works
A transformer's middle layers are dense activation vectors. Each neuron tends to fire for many unrelated things at once, a phenomenon called superposition: the model packs more concepts than it has neurons by encoding them as overlapping directions in a high-dimensional space. Reading the neurons one at a time gives noise.
Sparse autoencoders dissolve the superposition. The technique trains a much wider second network to reconstruct each activation vector using only a small number of "feature" directions at a time. After training, almost every feature corresponds to one concept that a human can name: the Golden Gate Bridge, sycophantic praise, code with security vulnerabilities, the abstract concept of inner conflict. Anthropic extracted roughly 34 million such features from Claude 3 Sonnet's middle layer. Around 1% were sharp enough that humans could label them confidently from examples alone.
The features are causal, not just correlational. Clamping a feature's value at 10x its normal maximum changes the model's behavior in predictable ways. Clamp the "transit infrastructure" feature and the model starts describing every input as a bridge or tunnel. Clamp the "secret plans" feature and it begins inventing conspiracies. This is the move that promoted interpretability from "we can correlate this neuron with this concept" to "we can edit the model's mind and watch the consequence."
Where it shows up
| Year | Lab | Artifact | What it showed |
|---|---|---|---|
| 2020 | OpenAI / Distill | Circuits thread | Vision networks contain car-detectors built from wheel + window + chrome features |
| 2022 | Anthropic | Toy Models of Superposition | Networks pack N concepts into D < N dimensions when concepts are sparse |
| 2023 | Anthropic | Towards Monosemanticity | Sparse autoencoders work on a tiny one-layer transformer |
| 2024 | Anthropic | Scaling Monosemanticity | Same technique scales to a frontier production model |
| 2024 | DeepMind | Gemma Scope | Open sparse autoencoders trained on every layer of Gemma 2 |
| 2024 | OpenAI | GPT-4 sparse autoencoder paper | 16 million features extracted; safety-relevant features identified |
Outside Anthropic and Google, the Apollo Research and EleutherAI groups have replicated the basic technique on open models. The neuronpedia.org project hosts a browsable library of extracted features for anyone to inspect.
What's contested
Three questions are genuinely open as of 2026.
First, do features generalize across architectures, or are they artifacts of a specific model's training run? A "Golden Gate Bridge" feature in Claude 3 Sonnet may not correspond to anything cleanly recoverable in GPT-4 or Llama 3. If features are model-specific, interpretability cannot become a general science of neural networks. It becomes a per-model exercise that has to be redone after every retrain.
Second, is the sparse-autoencoder framing the right one, or a convenient fiction? The technique imposes a sparsity prior. Features that emerge are the ones the prior was looking for. Critics including some of the original circuits researchers argue that real circuits in transformers are not cleanly sparse and that the technique smooths over the messier truth.
Third, how much of a model's behavior do the labeled features actually explain? Anthropic's own 2024 paper notes that the extracted features account for only a fraction of the variance in the activations they reconstruct. The unexplained remainder may contain the most safety-relevant computation, or it may be noise. Nobody knows.
Why this has to do with other realms
The bet that mechanistic interpretability is making is structurally similar to the bet concept neuroscience connectomics is making about the brain: that wiring + activations, mapped completely, will eventually yield understanding. Both fields are wagering that reductionism scales. Both are confronting the same uncomfortable possibility, which is that you can have a complete circuit diagram of a system and still not know what it is doing or why.
The interpretability community also borrows directly from physics. Superposition is named after the quantum concept for a reason, and the mathematical tools for unpacking it look like compressed sensing from signal processing. A field still arguing about its own foundations is reaching across the wall for tools that already work.
An open question
If a feature reliably fires for "deception" in one model and a structurally identical feature fires for "deception" in a different architecture trained on different data, does that count as evidence that deception is a natural concept in the universe of mind, or only evidence that two models trained on overlapping internet text learned the same prejudice?
Key sources
- Scaling Monosemanticity (Templeton, Conerly, Marcus et al., Anthropic, May 2024) — the Golden Gate paper. The single most-cited interpretability result of the decade so far.
- Toy Models of Superposition (Elhage et al., Anthropic, 2022) — the conceptual foundation. Why neurons polysemantically encode more than they should.
- Zoom In: An Introduction to Circuits (Olah et al., Distill, 2020) — the original case that neural networks contain inspectable subroutines, made with vision models.
- A Mathematical Framework for Transformer Circuits (Elhage et al., Anthropic, 2021) — the algebra that lets researchers reason about attention heads as named operations rather than opaque matrices.
- Neuronpedia.org (Bloom et al., ongoing) — the public catalog of extracted features, browseable per model and per layer.
Further reading
- Chris Olah on Mechanistic Interpretability (80,000 Hours podcast, 2024) — the clearest extended interview on why this research agenda exists and what it would mean if it failed.
- concept ai alignment — why anyone funds this work in the first place. Interpretability is alignment's most promising empirical wing.
- concept transformer architecture — the substrate. You cannot read what features mean without knowing what attention heads and MLP layers are doing.
- Anthropic's Interpretability Research page (anthropic.com/research) — the running list of papers from the lab that has invested most heavily in this direction.
- Apollo Research's writeups on deception circuits (2024-2025) — the most aggressive attempt to find safety-relevant features specifically rather than concepts in general.
Abhishek's take
I care about this because a buying-floor tool can be right for the wrong internal reason. When a model recommends a fabric move, I want to know whether it is seeing the QR tag, the sell-through curve, or just a familiar season name. Interpretability matters when one bad shortcut turns into a 100-day lead-time mistake.
See Also
- concept ai alignment (interpretability is the empirical arm of the alignment project)
- concept transformer architecture (the object being interpreted)
- concept neuroscience connectomics (the same reductionist bet, made on biological neurons instead of artificial ones)
- person chris olah (the researcher who turned a Distill blog into a field)
- concept superposition hypothesis (the specific theoretical claim that sparse autoencoders are designed to test)
- concept emergence in llms (what features may explain, if interpretability scales)