Does training's habit of stretching one learned idea to cover its neighbors slowly blur what a model's internal parts actually mean?

This explores whether the way models generalize as they learn, stretching a feature they've picked up for one thing to cover related things, causes meaning to drift: neurons that stop standing for one clear concept, or outputs that slide toward the generic. The corpus has no study that tackles this head-on. It does hold one sharp argument about recursive generalization and several separate findings about drift, and read together they fit into a useful picture.

The main entry point is the claim that polysemantic neurons (single units that fire for several unrelated-looking concepts) aren't mostly caused by superposition, the standard story that networks cram more features than they have neurons. Instead they come from branch-like recursive generalization: a neuron learns one concept, training extends it to a neighbouring one, then to a neighbour of that, until the unit's meaning has wandered Does recursive generalization explain polysemantic neurons better than superposition?. The surprising part is that this drift looks fixable. Architectural changes to InceptionV1 raised the share of interpretable neurons from almost none to 99-100%, which recasts messy neurons as a learning failure rather than an unavoidable cost of compression. A companion claim pushes further, saying most neurons in modern LLMs are already singly interpretable.

Drift during learning also shows up outside individual neurons, and it's measurable. When a model learns a new fact, the newly learned keyword can leak into unrelated contexts. That leakage can be predicted before training from how likely the model already found the word, with a threshold around 10^-3 and only about three exposures needed Can we predict keyword priming before learning happens?. That is generalization overshooting its target, and it's foreseeable. A quieter drift works at the level of word meaning. General words (hypernyms) are more common than specific ones, so a model that favours common phrasing slides steadily toward abstraction and wears away expert-level precision Does word frequency correlate with semantic abstraction?. Reinforcement learning shows a similar pull. When reward signals barely differ between attempts, the task signal fades and policies collapse into templates that ignore the input Why do language models collapse into generic templates?. In all three cases learning moves meaning toward whatever is broad, frequent, or already favoured.

Once meaning has settled, it resists correction. Strong associations from training can override what the prompt actually says, and prompting alone often can't fix this; you have to intervene directly in the model's internal representations Why do language models ignore information in their context?. That fits the view that model knowledge works less like stored records and more like a flow of activations, which makes drifted meanings hard to find and edit Do transformer models store knowledge or generate it continuously?.

One trap is worth flagging. Much of the corpus uses 'recursive' in a different sense: architectures that re-apply the same layers in a loop, which improve compositional generalization and state tracking Can models learn by looping instead of growing larger?, and recursive reasoning models whose gains come from variational training rather than added randomness Does adding randomness alone improve recursive reasoning models?. In those papers recursion is a tool that helps generalization. In Cammarata's argument, recursive generalization is a process that can make meaning drift. Same word, almost opposite roles. Keeping the two apart is probably the most useful thing to take away before going deeper.