Can sparse approximations reveal interpretable structure hidden in existing dense models?
This explores whether techniques that strip a trained dense model down to a sparse skeleton — pruning, sparse probing, removing 'unimportant' weights or tokens — actually expose interpretable machinery that was there all along, rather than building interpretability in from scratch.
This explores whether you can take an already-trained dense network and, by approximating it with something sparse, surface human-readable structure that the dense version hid. The corpus answers mostly yes — with an important caveat about what 'sparse' is doing the revealing.
The strongest evidence comes from pruning experiments. When you ablate or prune a trained network down to the minimal subnetwork needed for a task, compositional tasks turn out to be implemented by isolated, modular subnetworks — knock one out and only its corresponding function breaks Do neural networks naturally learn modular compositional structure?. That structure was always present in the dense weights; pruning is what makes it legible. The same logic plays out at the token level inside reasoning chains: a greedy likelihood-preserving pruning pass reveals that models internally rank tokens by functional role, preserving symbolic-computation steps while discarding grammar and filler — and models trained on those pruned chains actually outperform ones trained on frontier-model compressions Which tokens in reasoning chains actually matter most?. In both cases a sparse approximation acts like a developing agent on a latent image.
It's worth separating this from interpretability-by-design. Training transformers with sparse weights from the start produces clean, disentangled circuits where neurons map to simple concepts Can sparse weight training make neural networks interpretable by design? — but that's forcing modularity during training, not recovering it from an existing dense model. The pruning results are the more surprising claim: dense models trained normally already contain this modular skeleton, and pretraining makes it more consistent and reliable Do neural networks naturally learn modular compositional structure?. The structure is emergent, not imposed.
The caveat is sharp and easy to miss. A sparse or low-dimensional read of a model can also lie to you. Models can carry all the linearly decodable features a task needs while their internal organization is fundamentally fractured — broken in ways invisible to accuracy metrics but exposed under perturbation and distribution shift Can models be smart without organized internal structure?. So a sparse approximation that 'looks interpretable' isn't proof the underlying structure is sound; decodability is not the same as coherent internal organization. Relatedly, sparsity in these models isn't a fixed property you're simply uncovering — networks learn to be dense on familiar data and default to sparse representations on unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?, and they actively sparsify their activations as a stabilizing filter under hard, out-of-distribution tasks Do language models sparsify their activations under difficult tasks?.
The thing you might not have known you wanted to know: sparsity in these models wears two hats. It's both a *lens* we impose to read structure (pruning, sparse probing) and a *behavior* the model produces on its own (activations thinning out on unfamiliar inputs). When you prune to find a circuit, you're approximating; when the model sparsifies under OOD pressure, it's already telling you something about how it's organizing the problem. Read together, these notes suggest the most honest answer is: sparse approximations reliably reveal real modular structure — but you should verify that structure holds under stress before trusting that what you found is what the model is actually doing.
Sources 6 notes
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
Greedy likelihood-preserving pruning reveals six functional token categories; symbolic computation tokens are preferentially preserved while grammar and meta-discourse are pruned first. Student models trained on these pruned chains outperform those trained on frontier-model compression.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.