Do normal AI models already have readable insides, or does clarity only come from forcing them to be sparse?

This explores whether standard 'dense' networks are already readable inside, or whether you need special sparse training to get there. The corpus doesn't settle it. It does show the question is much more open than the usual story suggests. That story says dense neurons are a tangle. Each one fires for many unrelated things, so you either need sparse weights to force clean structure or a separate tool to untangle the model afterwards. Both approaches show up here. Training transformers with most weights set to zero produces small circuits where neurons map to simple concepts, though this hasn't yet scaled past tens of millions of parameters Can sparse weight training make neural networks interpretable by design?. Sparse autoencoders take the other route: they pull clean, abstract features out of a dense production model like Claude 3 Sonnet after training is done Can dictionary learning scale to production language models?. Both approaches assume the raw dense neurons are messy.

The strongest challenge to that assumption is Cammarata's argument Does recursive generalization explain polysemantic neurons better than superposition?. On this view, neurons that respond to many things aren't mainly the result of a network cramming too many features into too few neurons. They come from a fixable learning failure, a branch-like way of generalizing during training. When the architecture of the image model InceptionV1 was fixed, the share of neurons a person could read went from nearly zero to 99–100%. No sparse training was involved. If that holds for language models, then the messiness is a byproduct of current design choices, not something dense networks can't avoid.

Other notes suggest dense networks already organize themselves more than we assume. Pruning experiments find that standard networks put separate parts of a task into separate subnetworks. Disabling one subnetwork breaks only its own part of the task, and pretraining makes this split more reliable Do neural networks naturally learn modular compositional structure?. Dense models also become sparse on their own in a different sense: how many neurons are active at once. They use dense activity for familiar data and sparse activity for unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?, and their hidden states get sparser as tasks move further from what they were trained on Do language models sparsify their activations under difficult tasks?. So some sparsity may emerge on its own, with no one forcing it.

There's an important caution, though. A model can hold every feature a simple probe can find and still have a fractured, fragile internal organization that standard tests never reveal Can models be smart without organized internal structure?. Finding readable pieces is not the same as the whole model being readable. And 'sparse' means three different things in this corpus. It can mean weights set to zero, few neurons active at once, or attention that only looks at some of the input. The attention kind is about saving compute, not about interpretability Can sparse attention match dense models without retrofitting?. Mixing these up makes the debate look more settled than it is.

In short, there's growing evidence that dense models are more structured than the 'tangle' story claims. The clearest 'yes, already readable' result comes from an image model, not from LLMs at frontier scale. And there's a deeper question underneath: for some uses, like guiding scientific discovery, a model may not need to be readable at all Can opaque models guide discovery without needing interpretation?.