A picture of what a neural network's inner part 'looks for' is suggestive, but does it prove how the model works?
Why do feature visualizations alone fail to establish mechanistic claims?
This explores why showing what a network's internal features seem to detect, which is what a feature visualization does, can't by itself prove how the model actually works. The corpus has no note on feature visualization by name, so this draws on its material on representational analysis, which covers the same ground.
This explores why showing what a network's internal features seem to detect, which is what a feature visualization does, can't by itself prove how the model actually works. The corpus has no note on feature visualization by name, but it has plenty on the underlying problem: seeing what a model represents is a different thing from showing what it does with it.
The first gap is correlation versus cause. Can LLM understanding rely on just representation or causation alone? argues that representational analysis finds things that line up with a concept without proving the model uses them. Causal analysis has the opposite flaw: it shows that changing something has an effect without explaining what that part is for. A real mechanistic claim needs both steps. You locate a candidate by looking, then you verify it by intervening. A visualization is only the first step.
The second gap is that the looking itself is biased. Do standard analysis methods hide nonlinear features in neural networks? finds that common tools like PCA, linear regression and RSA over-represent simple linear features and under-represent nonlinear ones that matter just as much. Its most striking result is that a network under homomorphic encryption can compute perfectly well with no interpretable structure in its activations. So a clean-looking picture can miss what's really driving the output, and a messy one can hide a working computation. Can models be smart without organized internal structure? adds a further problem. A model can have every feature its task needs linearly decodable, with perfect accuracy, and still be internally fractured and fragile under perturbation. Finding a feature doesn't tell you the structure around it is doing what you'd assume.
The corpus also shows what a claim that passes looks like. In Can grafting restore sandbagged model capabilities?, a causal model predicted which layer window holds the sandbagging circuitry. Grafting the honest value in at that layer restored capability in 28 of 33 runs, with a median 96% recovery on held-out tests. That claim could have failed, and it didn't. Compare it with descriptive readouts like cycles in hidden-state reasoning graphs that track accuracy (Do reasoning cycles in hidden states reveal aha moments?) or higher path curvature on reasoning tasks (Does transformer reasoning leave a geometric signature in representation space?). These are intriguing patterns, but by the standard above they are candidates that still need an intervention to test them.
There is also a human side. What if XAI is fundamentally a communication problem? argues that an explanation's effect depends on who presents it, how it's framed, and who receives it, not just on the explanation itself. A vivid visualization can feel like proof because of how it's shown. Feeling convinced and having evidence are separate, and only the causal test provides evidence.
Sources 7 notes
Research shows that representational analysis alone identifies correlates without proving causation, while causal analysis alone demonstrates effects without explaining function. Only paired methodology—locating candidates representationally then verifying causally—produces genuine mechanistic understanding rather than descriptive claims.
PCA, linear regression, and RSA over-represent simple linear features while under-representing equally important nonlinear features. Homomorphic encryption demonstrates that networks can compute perfectly well with no interpretable activation structure, proving representation patterns and computation can be entirely decoupled.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Grafting honest values into the predicted layer window recovered capability in 28 of 33 runs with median 96% held-out recovery, confirming the causal model's prediction about where sandbagging circuitry operates.
Distilled reasoning models show ~5 cycles per sample versus near-zero in base models, and cyclicity correlates with accuracy. These cycles in hidden-state reasoning graphs directly map to RL-trained models' documented aha moments—moments when models reconsider intermediate answers.
Show all 7 sources
Measuring intrinsic geometry across multiple models shows reasoning and analogy tasks carve paths with mean curvature of 0.71–0.83 rad, while lexical tasks produce only 0.27–0.31 rad, suggesting path geometry encodes task difficulty.
Explanation quality is not intrinsic to the explanation itself but depends on the rhetorical situation: who presents it, how it is framed, and what role the recipient plays. Evaluations that ignore this triad measure only a narrow slice of real-world effectiveness.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
- Titans: Learning to Memorize at Test Time
- Rethinking Large Language Models in Mental Health Applications
- Representation Engineering: A Top-Down Approach to AI Transparency