How can neural networks be interpretable by design rather than post-hoc?
This explores how to build neural networks whose inner workings are legible from the start — through architecture and training choices — rather than reverse-engineering them after the fact.
This explores how to build neural networks whose inner workings are legible from the start, rather than reverse-engineering a trained black box. The corpus suggests the most direct route is changing how the network is built or trained so that structure falls out for free. The clearest example is training with weight sparsity: when you force most connections to zero, the network is pushed into compact, modular circuits where individual neurons line up with simple concepts, and ablation tests confirm those circuits actually do the work Can sparse weight training make neural networks interpretable by design?. A different architectural bet replaces the standard MLP's fixed activations and weight matrices with learnable spline functions on the edges — Kolmogorov-Arnold Networks — which not only shrink the model but make it possible to read off the mathematical relationships the network discovered Can learnable spline activations beat fixed MLP designs?.
There's an encouraging finding underneath these design moves: networks may already want to be modular. Pruning experiments show that even ordinary networks decompose compositional tasks into isolated subnetworks, where knocking out one piece affects only its corresponding function — and pretraining makes this modular structure more consistent and reliable Do neural networks naturally learn modular compositional structure?. So 'interpretable by design' can mean nudging a tendency the network already has, not imposing structure against its grain.
But the corpus also delivers a sobering reason why post-hoc interpretation is so fragile — and why design-time approaches matter. Two networks can produce identical outputs on every input while having radically different, internally broken representations; the 'fractured entangled representation' hypothesis shows SGD-trained models can pass every benchmark yet be tangled inside, vulnerable to perturbation and unable to transfer or recombine knowledge Can AI pass every test while understanding nothing? Can identical outputs hide broken internal representations? Can models be smart without organized internal structure?. Worse, the analysis tools we'd use to inspect a model after training are themselves biased: PCA, linear regression and RSA over-represent simple linear features and miss equally important nonlinear ones, and a homomorphic-encryption demonstration proves a network can compute perfectly with no interpretable activation structure at all Do standard analysis methods hide nonlinear features in neural networks?. If computation and readable representation can be fully decoupled, then waiting until after training to look inside is a losing game.
That reframes the whole question. Interpretability by design isn't only about adding sparsity or splines — it's about choosing training objectives that build coherent structure in the first place. Predicting your own latent representations (JEPA/data2vec style) provably recovers compositional hierarchy far more sample-efficiently than token prediction, because same-level latents are more correlated and the network is rewarded for organizing them Why is predicting latents more sample-efficient than tokens?. And energy-based transformers, which assign an energy to input-prediction pairs and minimize it at inference, build an explicit objective landscape you can reason about rather than an opaque feed-forward pass Can energy minimization unlock reasoning without domain-specific training?.
The thing you might not have expected: scale itself is a design lever. Compositional generalization — long thought to need symbolic machinery — emerges from plain MLPs once training data covers enough combinations, and when it works, the constituent parts become linearly decodable from the hidden activations Can neural networks learn compositional skills without symbolic mechanisms?. This sits in direct tension with the 'binding problem,' which argues networks structurally cannot bind distributed information into reusable compositional structures — though even that account concedes scale can partially conjure the needed representations into being Why do neural networks fail at compositional generalization?. The open frontier across all of this is the same one weight-sparsity hits: keeping the clean structure as you scale past tens of millions of parameters.
Sources 11 notes
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Kolmogorov-Arnold Networks replace MLPs' fixed activations and linear weights with learnable univariate splines on edges, achieving better accuracy with smaller models, faster neural scaling laws, and built-in interpretability for discovering mathematical laws.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
The Fractured Entangled Representation hypothesis shows that SGD-trained networks can produce identical outputs across all inputs while maintaining radically different internal representations. Standard benchmarks cannot detect this structural difference.
Networks trained with SGD reproduce outputs perfectly while having radically different internal structure than evolved networks, with weight perturbations revealing fractured, entangled representations that prevent transfer to novel contexts or creative recombination.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
PCA, linear regression, and RSA over-represent simple linear features while under-representing equally important nonlinear features. Homomorphic encryption demonstrates that networks can compute perfectly well with no interpretable activation structure, proving representation patterns and computation can be entirely decoupled.
A formal sample-complexity analysis proves latent-level self-supervision (data2vec/JEPA style) recovers compositional structure with samples constant in hierarchy depth, while token-level learning requires exponential samples—because same-level latents are far more correlated than raw tokens.
Energy-Based Transformers assign energy values to input-prediction pairs and use gradient descent minimization for inference, yielding 35% higher training scaling rates and 29% more inference-compute gains than Transformer++, while generalizing better on out-of-distribution data without domain-specific scaffolding.
Standard MLPs achieve compositional generalization through data and model scaling alone, without architectural modifications, provided the training distribution sufficiently covers combinations of task modules. Linear decodability of constituents from hidden activations reliably predicts success.
Greff et al. argue that neural networks cannot dynamically bind distributed information into compositional structures due to three failures: segregating entities from inputs, maintaining representational separation, and reusing learned structure in novel combinations. Scaling can partially overcome this by enabling compositional representations to emerge.