Claims that 'most neurons are interpretable' jumped from near-zero to 97% in two years — did the AI change, or just how we measure it?
How much do measurement thresholds affect estimates of neuron interpretability rates?
This explores how much the reported share of 'interpretable' neurons depends on how researchers define and test interpretability, rather than on how the networks actually work.
This explores whether claims like 'most neurons are interpretable' measure the networks or the yardstick. The short answer from the corpus: the yardstick seems to matter a lot, though no note here runs a formal sensitivity test that varies one threshold and watches the number move. The clearest evidence is indirect, and it is striking. Cammarata reports that roughly 90–97% of neurons in today's largest language models do one main job, up from near-zero on 2023 benchmarks. He credits better tools, not changes to the models How interpretable are neurons in today's large language models?. If the models didn't change much and the number jumped from almost nothing to almost everything, then much of that jump comes from measurement. Look closely at his definition too: 'one main job *or related variations*.' How loosely you read 'related' is a threshold in itself, and it decides whether a neuron counts as clean or as polysemantic (responding to several unrelated things).
Cammarata's companion argument shows how a definition can change what you think you're seeing. He claims many neurons that look polysemantic are really doing branch-like generalization: one concept that splits into related sub-cases as training goes on Does recursive generalization explain polysemantic neurons better than superposition?. Under a strict 'one concept only' rule, those neurons fail. Under a 'one family of concepts' rule, they pass. His InceptionV1 result, where architectural fixes pushed interpretable neurons from nearly zero to 99–100%, suggests some of the change is real. Still, the two readings of the same neuron would produce very different headline rates.
Other parts of the collection show that 'interpretable' isn't one bar. It's a stack of bars, and the rate depends on which one you pick. One line of work says a neuron's apparent meaning (its representation) is only a correlation until you show that intervening on it actually changes behavior (causal testing) Can LLM understanding rely on just representation or causation alone?. A rate based on 'this neuron fires on X' will run higher than one that requires 'removing this neuron breaks X.' The program-synthesis work makes the trade-off concrete. Synthesized code can match attention-head behavior at 99% fidelity, but swapping in programs only preserves the model's question-answering for the best-fitting 30–40% of heads Can executable programs replace transformer attention heads?. Choose 'explainable in words' as your bar and the number looks high. Choose 'replaceable by an exact program without loss' and it shrinks to under half.
The input data matters too, and this is easy to overlook. Networks develop dense, overlapping activations on familiar training data and sparse ones on unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?. So the same neuron can look clean or tangled depending on which examples you probe it with. The fractured-representation research adds a warning on the other side. Networks can produce identical outputs while hiding very different, tangled internal structure that standard tests don't catch Can identical outputs hide broken internal representations?. A neuron that passes an activation-based check might still be part of a fractured system. Weight-sparsity training offers a contrast: it builds networks where interpretability is enforced by design and checked by ablation (switching parts off to see what breaks), so the bar is set before measurement rather than chosen afterward Can sparse weight training make neural networks interpretable by design?.
The takeaway you may not have expected: when you see an interpretability percentage, ask three things. What counted as 'one job'? Was the claim checked causally? What data were the neurons probed on? The collection suggests these choices can move the answer from near zero to near total. A direct study that varies thresholds on one model would settle how much, and the corpus doesn't have one yet.
Sources 7 notes
Cammarata's audits of models up to trillions of parameters found roughly 90-97% of neurons perform one main job or related variations, a dramatic shift from 2023 benchmarks showing near-zero interpretability. He attributes this to improved tools, not architectural changes.
Cammarata argues polysemantic neurons emerge from branch-like recursive generalization during training rather than arbitrary feature packing. Architectural fixes to InceptionV1 pushed interpretable neurons from nearly zero to 99-100%, suggesting the issue was a fixable learning failure.
Research shows that representational analysis alone identifies correlates without proving causation, while causal analysis alone demonstrates effects without explaining function. Only paired methodology—locating candidates representationally then verifying causally—produces genuine mechanistic understanding rather than descriptive claims.
Program synthesis recovers executable code matching 99% of attention head behavior. Substituting the best-fit 30–40% of heads with their synthesized programs preserves QA ability, offering formal, testable interpretability instead of natural-language summaries.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
Show all 7 sources
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
- Weight-sparse transformers have interpretable circuits
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Open Problems in Mechanistic Interpretability
- Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
- Neural Networks Are Already Clean
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Break It Down: Evidence for Structural Compositionality in Neural Networks