Why can we follow some AI circuits but not others — and is that clarity natural, or something training must build in?
What makes some internal circuits more interpretable than others?
This explores what about a network's internal wiring lets a human read and follow a circuit, and whether that readability is a natural property of networks or something training has to build in.
This explores what makes a circuit inside a network readable to humans, and whether readability is natural or has to be built in. The corpus's sharpest answer is that interpretability depends heavily on how the network was trained. It isn't a given. When researchers train transformers with sparse weights, meaning most connections are forced toward zero, the network is pushed to be modular. The result is compact circuits where individual neurons line up with simple concepts and the connections between them are clear. Ablation tests show these circuits are both necessary and sufficient for the task Can sparse weight training make neural networks interpretable by design?. The catch is that this has only been shown up to tens of millions of parameters, and keeping it readable at larger scale is unsolved.
Ordinary training gives you the opposite by default. Networks trained with standard SGD can match the outputs of a cleaner network exactly while being 'fractured and entangled' inside. Every feature the task needs can be decoded, but the organization is broken, so there is no tidy circuit to find Can identical outputs hide broken internal representations? Can models be smart without organized internal structure?. Perturbing the weights shows that these fragments don't transfer to new contexts or recombine creatively. Because benchmarks only check outputs, they can't tell the difference, which is why one piece calls such a model an imposter that passes every test Can AI pass every test while understanding nothing?. Some circuits are hard to read because there isn't a clean one there in the first place.
Readable also doesn't mean in charge. The corpus's overview of mechanistic interpretability says internal organization and external performance are surprisingly decoupled, and a circuit that looks neat may not causally drive the output What actually happens inside large language models?. That gives a two-part test for calling a circuit interpretable: can you read it, and does removing it break the behavior? The sparse-weight work is notable because it runs the second test, not just the first.
Some circuits do turn out clean, and one example shows how much the training recipe matters. Models can detect steering vectors injected into their activations through a two-stage circuit. Evidence-carrier features in early layers suppress gate features that would otherwise default to denial How do language models detect injected steering vectors internally?. It has a clear division of labor, and it appears with DPO but not SFT. Safety training then cuts detection from 63.8% to 10.8%, so training choices decide whether a legible circuit exists at all. Two other approaches skip the wiring diagram and read what is represented instead. Representation engineering treats concepts like truthfulness as linear directions in activation space, extracts them with 90%+ accuracy, and steers behavior with them Can high-level concepts replace circuit-level analysis in AI?. LatentQA trains a decoder that answers plain-language questions about activations Can we decode what LLM activations really represent in language?. A third angle reads the shape of activity over time: reasoning cycles in hidden states, about five per sample in distilled reasoning models versus near zero in base models, line up with 'aha moments' Do reasoning cycles in hidden states reveal aha moments?.
The corpus suggests the tidiest circuits are the ones training forced to be tidy. Whether that tidiness survives at frontier scale is still open.
Sources 9 notes
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Research shows identical accuracy can mask fundamentally different or corrupted internal representations, and mechanistically interpretable circuits may not causally drive outputs. Internal organization and external performance follow distinct paths.
Show all 9 sources
Contrastive preference optimization trains evidence-carrier features in early layers to suppress gate features that default to denial, enabling near-perfect detection of internal perturbations. Safety training actively suppresses this capability, reducing detection from 63.8% to 10.8%.
Treating high-level concepts as linear directions in activation space (the Hopfieldian view) provides a scalable alternative to circuit-level mechanistic interpretability. Experimental results show 90%+ accuracy in extracting concepts like truthfulness and demonstrate causal control through vector manipulation.
LatentQA trains a decoder to answer natural language questions about LLM activations, enabling both interpretability (understanding what activations encode) and controllability (steering them via gradient descent). Critical design choices—activation masking, diverse training data, and faithful completions—proved essential for generalization.
Distilled reasoning models show ~5 cycles per sample versus near-zero in base models, and cyclicity correlates with accuracy. These cycles in hidden-state reasoning graphs directly map to RL-trained models' documented aha moments—moments when models reconsider intermediate answers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis
- Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMs
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Break It Down: Evidence for Structural Compositionality in Neural Networks
- Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
- Emergent Introspective Awareness in Large Language Models
- Open Problems in Mechanistic Interpretability
- Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control