Can you train an AI to be sparse enough for humans to read its 'thoughts' without making it dumber?

This explores whether sparsity can be built into training so a model is readable by design and still performs well. The short answer from the corpus is: partly, and only at small scale so far. The most direct evidence comes from work that trains transformers with most of their weights forced to zero Can sparse weight training make neural networks interpretable by design?. Under that constraint, the model breaks into compact circuits where individual neurons map to simple concepts and the connections between them can be traced by hand. Ablation tests show these circuits are both necessary and sufficient for the task, so they are the real mechanism behind the behavior and not just a convenient story. The catch is that no one has yet kept this level of readability past a few tens of millions of parameters. That is tiny next to frontier models, so the capability question is still open.

At production scale, interpretability work has mostly taken another route: leave the model dense and capable, then apply sparsity afterward to read it. Sparse autoencoders trained on Claude 3 Sonnet's internals recover abstract, multilingual features that both track and causally steer the model's behavior Can dictionary learning scale to production language models?. This avoids the trade-off rather than solving it. Capability is untouched because the sparse layer is a lens on the model, not part of it. The price is that you get a translation of the model's reasoning, not a model whose reasoning is readable directly. That difference matters, because a feature that only correlates with a behavior doesn't explain the behavior until an intervention shows it causes it Can LLM understanding rely on just representation or causation alone?. Built-in sparsity has an edge here: its ablation evidence comes with the design.

There is a less obvious reason the trade-off might be real. Models already move between sparse and dense states on their own, and the pattern seems tied to what they know. During pretraining, networks build dense activations for familiar data and fall back to sparse ones for unfamiliar inputs Is representational sparsity learned or intrinsic to neural networks?. When tasks get harder or less familiar, LLM hidden states become noticeably sparser, which seems to act as a stabilizing filter rather than a breakdown Do language models sparsify their activations under difficult tasks?. Together these suggest that density is partly how a model stores what it knows well. If so, forcing sparsity everywhere could limit the dense, overlapping representations that carry well-learned knowledge, which may be why readable-by-design models have been hard to scale.

One result gives some grounds for optimism, though it comes from a different goal. MiniMax's sparse attention matches full attention at 109B parameters while cutting attention compute by 28.4× at a 1M-token context Can sparse attention match dense models without retrofitting?. The key was training the sparsity in from the start instead of adding it later. That work targets efficiency, not interpretability, and attention sparsity is not the same as sparse weights. Still, it suggests sparsity built in during pretraining can hold up at frontier scale where retrofitted sparsity tends to fail. The corpus doesn't yet have a paper that achieves both high interpretability and frontier capability. A useful open question is whether combining native sparse training with the circuit-level approach could get past the tens-of-millions ceiling.