Do neurons that respond to several unrelated things prove networks are forced to overlap concepts — or is that just sloppy training?

This explores whether the superposition hypothesis is the only good explanation for 'polysemantic' neurons, meaning single neurons that fire for several unrelated concepts. On the superposition view, a network has more concepts to represent than it has neurons, so it overlaps them, and some mixing can't be avoided. The collection's sharpest challenge to that view says the mixing is mostly a fixable training failure, not a law of compression. Cammarata argues that polysemantic neurons come from branch-like 'recursive generalization' during training, where a neuron learns one thing and later stretches to cover related or unrelated cases, rather than from deliberate packing Does recursive generalization explain polysemantic neurons better than superposition?. The evidence that stands out: architectural fixes to the image model InceptionV1 raised the share of interpretable neurons from almost none to 99–100%. If superposition were truly necessary, fixes like that shouldn't work so well.

Other notes approach the same question from different directions and point the same way: neural networks seem to be more orderly inside than the superposition story assumes. When researchers prune networks trained on tasks built from parts, they find separate subnetworks for each part, and knocking one out breaks only its own function Do neural networks naturally learn modular compositional structure?. Training transformers with sparse weights, so each neuron has only a few connections, produces small circuits whose neurons map to simple concepts Can sparse weight training make neural networks interpretable by design?. That second result cuts both ways, though. It shows clean representations are possible when you force them, and it also hints that standard training doesn't produce them by default. The authors also say they can't yet keep this interpretability above tens of millions of parameters, which is the scale where superposition supporters would expect overlap to matter most.

A quieter finding complicates the picture: how dense a model's representations are seems to depend on familiarity. Models use dense activations for inputs that resemble their training data and fall back to sparse ones for unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?. So how much concepts get crowded together may not be a fixed property of the network. It may change with what the model is looking at, which a single 'networks compress features' story doesn't capture well.

The work on 'fractured entangled representations' is the strongest warning against either clean answer. Networks trained the standard way can produce exactly the same outputs as networks with very different, better-organized internals Can identical outputs hide broken internal representations?, and ordinary benchmarks can't tell them apart Can AI pass every test while understanding nothing?. In other words, a network's behavior doesn't settle what is happening inside it. Messy internals might come from compression, from how the network happened to learn, or from both, and output-level tests alone won't tell you which.

One gap you should know about: this collection has the challengers to superposition, but no note that makes the case for it, such as the toy-model or sparse-autoencoder work behind the idea. So the fair takeaway is that superposition is not proven necessary, and there is credible evidence that much of the mixing can be trained away. It is not that superposition has been ruled out. There's also a philosophical point that may change why the question matters to you: some argue that opaque models can still guide real scientific discovery, because what gets justified is the resulting theory, not the model's internals Can opaque models guide discovery without needing interpretation?. How much you need to settle this debate depends on whether you want to understand the model or only to use what it finds.