What makes a neural network circuit actually interpretable to humans?
This explores what actually has to be true of a neural circuit before a human can claim to understand it — not just whether it looks tidy, but whether the tidiness maps onto something real about how the model computes.
This explores what actually has to be true of a neural circuit before a human can claim to understand it — and the corpus splits the answer into two demands that pull against each other. The first is structural: a circuit is interpretable when it's *modular* and *disentangled*, meaning a small, isolatable cluster of neurons corresponds to a recognizable concept and connects cleanly to others. Training transformers with sparse weights forces exactly this, producing compact circuits where neurons stand for simple concepts Can sparse weight training make neural networks interpretable by design?. And networks don't only get this when forced — pruning experiments show they sometimes carve compositional tasks into isolated subnetworks on their own, with pretraining making that modular structure far more reliable Do neural networks naturally learn modular compositional structure?.
But clean structure is necessary, not sufficient — and the sharper lesson in this collection is the second demand: the circuit has to *causally drive the output*, not merely correlate with it. Several notes converge on an uncomfortable finding: internal structure and external behavior are decoupled. Two models can score identically while computing through radically different internal mechanisms, and a mechanism that looks interpretable may not actually be the thing producing the answer What actually happens inside the minds of language models?. The 'Fractured Entangled Representation' work makes this vivid — networks can reproduce outputs perfectly while their internals are a tangled mess that breaks the moment you perturb a weight or move to a novel context Can identical outputs hide broken internal representations?. A model can pass every benchmark and still have incoherent internal organization, which means the test you trusted can't see the difference Can AI pass every test while understanding nothing?. So the real interpretability bar isn't 'does this story sound plausible' but 'does ablating this circuit actually change the behavior it supposedly controls' — which is why the sparse-circuit work leans on ablation studies to prove necessity and sufficiency Can sparse weight training make neural networks interpretable by design? What actually happens inside a language model?.
Here's the part you might not expect: our analysis tools quietly cheat in favor of legibility. The standard methods — PCA, linear regression, RSA — systematically over-surface simple linear features and miss equally important nonlinear ones. The clincher is a homomorphic-encryption demonstration showing a network can compute perfectly with *no* interpretable activation structure at all, proving that 'looks interpretable' and 'is doing the computation' can be fully unhooked Do standard analysis methods hide nonlinear features in neural networks?. In other words, some of the interpretability we celebrate may be an artifact of looking only where the light is good.
Given how hard circuit-level analysis is to scale and trust, part of the corpus argues for going *over* the circuit rather than into it. Representation engineering treats high-level concepts like 'truthfulness' as linear directions in activation space and gets 90%+ extraction accuracy plus causal control by nudging those vectors — a top-down route that sidesteps wiring up individual neurons Can high-level concepts replace circuit-level analysis in AI?. A related move uses an LLM as a surrogate that explains another model by aligning to both its outputs and its internal embeddings, deliberately balancing faithfulness-to-the-model against readability-to-a-human Can LLMs explain recommenders by mimicking their internal states?. Both treat interpretability as a translation problem, not a dissection problem.
Underneath all of it sits a quieter claim worth taking home: interpretability is ultimately for *human oversight*, so explanations have to live in forms humans can reason about and validate — independent of whether the AI can explain itself Can humans understand deep learning before AI does?. Putting the threads together, a circuit is genuinely interpretable when it satisfies four things at once: it's modular, it maps to concepts a person recognizes, ablating it provably changes behavior, and the human-facing explanation survives contact with how the model actually computes rather than how our biased tools prefer to draw it.
Sources 10 notes
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
LLMs can achieve identical accuracy while maintaining radically different internal representations, and mechanisms that appear interpretable may not causally drive outputs. This decoupling means performance metrics alone mask crucial differences in how models actually work.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Research shows that LLMs can achieve the same output through different internal mechanisms, and improvements in one dimension like accuracy reliably degrade others like faithfulness and calibration. Internal structure matters even when behavior appears identical.
PCA, linear regression, and RSA over-represent simple linear features while under-representing equally important nonlinear features. Homomorphic encryption demonstrates that networks can compute perfectly well with no interpretable activation structure, proving representation patterns and computation can be entirely decoupled.
Treating high-level concepts as linear directions in activation space (the Hopfieldian view) provides a scalable alternative to circuit-level mechanistic interpretability. Experimental results show 90%+ accuracy in extracting concepts like truthfulness and demonstrate causal control through vector manipulation.
RecExplainer trains LLMs via three alignment methods: behavior (mimicking outputs), intention (incorporating neural embeddings), and hybrid (combining both). The hybrid approach produces explanations that are simultaneously faithful to the target model and intelligible to users by balancing internal-state inspection with human-readable reasoning.
Deep learning theory must be developed in forms humans can reason about and evaluate, because human oversight of AI systems depends on frameworks for identifying failure modes and validating explanations—not on whether AI can self-explain.