If you force most of an AI's connections to zero, can a human finally read how it thinks?

This explores whether forcing a network's weights to be mostly zero makes its inner workings readable to humans, and how that differs from the other kinds of "sparse" in the collection. On small models, the answer is yes. Training transformers with sparse weights means each neuron connects to only a few others. The result is compact circuits where neurons correspond to simple concepts and the connections between them are clear Can sparse weight training make neural networks interpretable by design?. Ablation tests show these circuits are both necessary and sufficient for the task: knock them out and performance fails, keep only them and it holds. The catch is scale. This has only been shown up to tens of millions of parameters, and keeping the circuits readable in larger models is unsolved.

The reason sparsity helps is forced modularity. When a neuron has few connections, the network can't smear one concept across everything, so the pieces stay separate. This matters because ordinary dense training doesn't guarantee that structure. A model can contain every feature it needs, perfectly decodable, and still be organized in a fractured way that only shows up under perturbation or distribution shift. Standard accuracy numbers never reveal it Can models be smart without organized internal structure?. Sparse-by-design is a bet that clean structure has to be built in rather than hoped for. The collection doesn't test the two head to head, so that link is my reading, not a finding.

The other route leaves the model's weights alone and puts the sparsity in the readout. Sparse autoencoders (dictionary learning) pull interpretable, abstract, multilingual features out of Claude 3 Sonnet, and those features causally change the model's behavior when nudged Can dictionary learning scale to production language models?. So there are two trade-offs. Sparse weights make the model itself legible but haven't been shown at scale. Sparse feature dictionaries work at production scale, but the model underneath stays dense and tangled, and you read it through a translation layer.

Two other kinds of sparsity in the collection are easy to confuse with this one. Sparse *activations* are something models do rather than something imposed. Hidden states get markedly sparser as tasks become unfamiliar or harder, which works as an adaptive filter that stabilizes performance Do language models sparsify their activations under difficult tasks?. Pretraining teaches this pattern: dense representations for familiar data, sparse for unfamiliar Is representational sparsity learned or intrinsic to neural networks?. That makes activation sparsity a possible signal of what the model knows well, not a route to readable circuits. Sparse *attention* is about compute. It cuts per-token attention cost 28.4× at 1M context while matching dense models at 109B Can sparse attention match dense models without retrofitting?. It makes no claim about legibility. It does echo the sparse-weights result in one way: sparsity works when it's trained in from the start, not retrofitted afterward.

How would you know a sparse circuit is real rather than a plausible story? The collection's answer is to pair two kinds of evidence: find candidates by looking at representations, then verify them by intervening Can LLM understanding rely on just representation or causation alone?. The ablation tests on sparse circuits do exactly that. The open question is whether sparse weights can reach the scale where the dictionary-learning work already operates.