Why does weight sparsity reduce superposition and force disentangled representations?
This explores why constraining a network's weights to be mostly zero pushes features to stop sharing the same neurons (superposition) and instead occupy separate, readable circuits — and what the corpus reveals about the mechanism behind that.
This explores why forcing most of a network's weights to zero breaks superposition — the trick where a model crams many features into the same overlapping neurons — and pushes it toward separated, one-concept-per-circuit representations. The most direct answer in the collection is that sparsity removes the connective tissue superposition depends on: when a neuron is only allowed a handful of incoming and outgoing weights, it can no longer participate in dozens of features at once, so the network is forced to commit each neuron to a narrow job. Can sparse weight training make neural networks interpretable by design? shows transformers trained this way grow compact circuits where individual neurons map to simple concepts with clear wiring, and ablations confirm those circuits are both necessary and sufficient for the task — the disentanglement is real structure, not a visualization artifact.
What makes this more than a one-paper finding is that the same pressure shows up wherever the corpus studies modularity. Pruning experiments in Do neural networks naturally learn modular compositional structure? find that networks already *want* to isolate compositional subroutines into separate subnetworks — knocking out one subnetwork only damages its corresponding function. Weight sparsity can be read as turning that latent tendency into an enforced constraint: instead of hoping modularity emerges, you make dense entanglement structurally impossible. The recommender-systems note Can a linear model beat deep collaborative filtering? tells the same story from a different field — a zero-diagonal constraint forces the model to route every prediction through genuine item relationships rather than shortcuts, and the authors conclude that structural bias mattered more than raw capacity. Across vision, language, and recommendation, the lesson rhymes: the right constraint does more disentangling work than more parameters.
The reason this matters — the thing you might not have known you wanted to know — is that disentanglement and *performance* are nearly invisible to each other. Can models be smart without organized internal structure? shows two models can score identically while one is cleanly organized and the other is internally fractured, with all the right features linearly decodable on top of broken internal structure. That fractured organization stays hidden until distribution shift or perturbation breaks it. Superposition is exactly the kind of entanglement that boosts efficiency without showing up on the scoreboard, which is why you have to *impose* sparsity rather than wait for accuracy to reward it — accuracy never will.
There's a deeper framing worth pulling in. Why do neural networks fail at compositional generalization? argues that neural networks struggle precisely because they can't keep distributed information about different entities separated — features bleed into one another. Superposition is the efficient-but-costly version of that bleeding. Seen this way, weight sparsity is one concrete lever on the binding problem: by physically separating which neurons can talk to which, it forces the representational segregation Greff et al. say is otherwise hard to maintain. The architecture work in Does depth matter more than width for tiny language models? hints at a complementary lever — composing concepts through depth rather than spreading them across width — suggesting sparsity isn't the only structural route to disentanglement, just the most surgical one we have.
One honest limit: the corpus demonstrates *that* sparsity disentangles and shows the circuits it produces, but it's lighter on a from-scratch mechanistic account of superposition itself, and Can sparse weight training make neural networks interpretable by design? flags that keeping this interpretability past tens of millions of parameters is still unsolved. So the clean circuits are real but, for now, small.
Sources 6 notes
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.
Pruning experiments reveal that neural networks implement compositional subroutines in isolated subnetworks, with ablations affecting only their corresponding function. Pretraining substantially increases the consistency and reliability of this modular structure across architectures and domains.
ESLER, a single-layer linear autoencoder constrained so items cannot predict themselves, outperforms most deep CF models. The constraint forces prediction through item relationships, and negative weights encoding anti-affinity prove essential—structural bias matters more than model capacity.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Greff et al. argue that neural networks cannot dynamically bind distributed information into compositional structures due to three failures: segregating entities from inputs, maintaining representational separation, and reusing learned structure in novel combinations. Scaling can partially overcome this by enabling compositional representations to emerge.
MobileLLM shows deep-and-thin architectures yield 2.7–4.3% accuracy gains over balanced designs at 125M–350M scale by composing abstract concepts through layers rather than spreading parameters across width.