AI 101 — What is mechanistic interpretability?

Mechanistic interpretability is the work of reverse-engineering what actually happens inside a neural network — finding the specific internal parts responsible for a specific answer, rather than inferring everything from the model's behaviour. "Mechanistic" is the load-bearing word: the deliverable is a mechanism, a step-by-step account of the computation, not a description of what the model tends to do.

Why it matters right now

The cheapest way anyone audits a frontier model is by reading its chain of thought — the scratchpad where it works through a problem in English. That window is closing. OpenAI's system card for GPT-6 Astra, its first model to reach the "Critical" level of cyber capability, reports a "substantial decrease in chain-of-thought monitorability." In September, Google DeepMind's Rohin Shah and Anca Dragan published a public case for reasoning transparency, arguing that efficiency pressure is pushing labs toward architectures whose reasoning is written in internal numbers instead of sentences — a worry we reported on in Astra's hidden reasoning loop is the real story, not Astra. If the readable reasoning goes away, the labs need a second way to see inside. This is that second way.

It matters for a second reason, closer to the money. Models are now trained with reinforcement learning on tasks where the reward is a proxy for what we actually want, and proxies get gamed. Explaining "the model did something we didn't intend" requires knowing why, and mechanistic interpretability is the only route to that answer that doesn't take the model's word for it.

The mental model

A model's computation happens in its activations — enormous lists of numbers that change as text flows through. The obvious unit to study would be the neuron, and that mostly fails: individual neurons fire on unrelated things, a problem called polysemanticity. The leading explanation is superposition: a model has far more concepts to represent than it has neurons, so it stores concepts as near-orthogonal directions that overlap. Think of a dictionary with more words than there is paper, so several definitions are printed faintly on top of each other.

The workflow that emerged is two-step. First find the features — interpretable building blocks. Researchers train a small extra network called a sparse autoencoder to pull activations apart into directions that are only occasionally active, which turns out to correspond to concepts a human can name. Anthropic did this on the middle layer of Claude 3 Sonnet and reported tens of millions of features, including one for the Golden Gate Bridge that fires whether the bridge is mentioned in English, Japanese or Russian, and even generalises to photographs. Then map the circuits — which features feed which. Anthropic's attribution graphs trace the steps behind a single output in Claude 3.5 Haiku; OpenAI took the opposite route in a November 2025 paper, forcing most weights to zero during training so each neuron keeps only a handful of connections, then pruning a model down to the circuit that decides whether a Python string should be closed with a single or a double quote — five residual channels, two neurons in one layer, one attention channel pair in another.

The car analogy

Behavioural testing is driving a car around the block. You learn what it does — how it accelerates, when the check-engine light comes on, how much fuel it burns — and for a lot of purposes that is genuinely enough. Mechanistic interpretability is opening the hood and tracing which wire carries which signal, with two complications. The hood is welded shut, and there are billions of wires. Feature work is learning to recognise the wiring looms: the bundles that light up together for one concept, wherever that concept appears. Circuit work is following a single path from the ignition key through the loom to the fuel injector, and then checking your map by cutting a wire and seeing whether the engine behaves the way the map predicted.

Common misconceptions

"It's just explainability." Standard explainability tools score which parts of your input mattered to an output. Mechanistic interpretability makes a claim about the model's internal computation, then tests that claim causally. The demonstration everyone remembers: clamping the Golden Gate Bridge feature to ten times its maximum activation made Claude steer almost every answer toward the bridge, including recommending the toll as a good use of $10. That is evidence of a mechanism, not a correlation.

"The chain of thought is the model's source code." No — it is text the model was trained to produce. OpenAI's own framing separates the two: chain-of-thought interpretability reads what the model writes down, while mechanistic interpretability tries to reconstruct the computation itself. The scratchpad is useful and it is not a transcript. Our AI 101 — What is chain of thought? explainer covers why the written reasoning is best understood as a performance the model learned to give.

"Open weights means interpretable." The numbers are public and unreadable. Open weights let you run, fine-tune and study a model; they don't tell you what any of the billions of values means.

"This is basically solved." The primary sources are candid about the ceiling. OpenAI states its sparse models are far smaller than frontier systems and that large parts of their computation remain uninterpreted, and its paper flags that scaling sparse models past tens of millions of nonzero parameters while keeping the circuits readable is still unsolved. Anthropic's attribution graphs explain one prompt at a time — a graph for "Dallas" doesn't guarantee anything about "Houston." And because these methods study an approximation of the model rather than the model itself, every claim has to survive the perturbation test.

Where to learn more

Anthropic's Transformer Circuits site is the field's public lab notebook — Scaling Monosemanticity for features, Circuit Tracing and On the Biology of a Large Language Model for circuits. OpenAI's sparse-circuits post is the clearest statement of the alternative bet: build models that are readable by design. And the DeepMind Institute essay is worth reading for why this is now a policy question and not just a research one.

Related reading: our coverage of Sparse weight decomposition — LLM circuits from under 1% of the data, where researchers argue the circuits are already sitting in the weights.

If a model's reasoning stops being readable in English, is reverse-engineering its internal circuits a real substitute — or a slower way of guessing? Tell us in the comments.