Related research

Neural networks are not already clean

https://www.lesswrong.com/posts/Dkh7pXKjoNWkkxQDz/neural-networks-are-not-already-clean

The assumption that modern language models arrive at interpretability through training has framed much recent work, but emerging evidence challenges whether clean, atomic neuron representations arise organically from standard learning dynamics. This paper joins a growing tension in interpretability research: while some preliminary studies suggest neurons already have interpretable single functions, others point to persistent polysemy and entanglement—raising the question of whether structural constraints during training or layered analysis methods are necessary preconditions for interpretability rather than happy accidents of scale. If neural networks genuinely fail to self-organize into clean representations, what does that imply for post-hoc interpretation efforts—and for how we should architect models from the ground up?

Lines of inquiry this post opens

Not questions with answers — ways of approaching this material across the collection.

What determines how neural networks organize their internal knowledge representations?

How do mechanistic features and representations drive model behavior?

How can parameter sparsity improve interpretability while maintaining model capability?

Does learning articulatory physics explain speech communication superiority?

Can spectral eigenvector ordering serve as a model-agnostic interpretability probe?

Adjacent research

Synthesis notes nearest this post — click to read.

Does recursive generalization explain polysemantic neurons better than superposition? How interpretable are neurons in today's large language models? Can cognitive science methods unlock how LLMs actually work? Can sparse weight training make neural networks interpretable by design? Can learnable spline activations beat fixed MLP designs? Can opaque models guide discovery without needing interpretation? Can we probe foundation models without any input data? Can humans learn chess concepts that AlphaZero discovered alone?

Topics touched