AI attention can't assign zero weight to any word — so irrelevant context always bleeds through, creating a noise floor.
Why does standard softmax spread attention across irrelevant tokens?
This explores why the softmax function at the heart of transformer attention can't fully zero out irrelevant tokens — and what that costs.
This explores why standard softmax attention leaks probability onto tokens that don't matter — and the corpus frames it as a structural property, not a bug you can prompt away. The mechanical reason is simple: softmax normalizes a full row of scores into a distribution that must sum to one, and it can never assign exactly zero. So every token gets a sliver of weight, and across a long context those slivers accumulate into a persistent 'noise floor' that dilutes the real signal. The most direct treatment of this is differential attention, which computes attention as the *difference* of two softmax maps so the shared noise cancels — like noise-cancelling headphones for the attention pattern Can subtracting attention maps remove noise from context?.
But the spread isn't uniformly random — it's biased. Soft attention systematically over-weights tokens that are repeated or contextually prominent regardless of whether they're relevant, creating a feedback loop that amplifies framing and even helps explain sycophancy Does transformer attention architecture inherently favor repeated content?. Underneath, models seem to fight softmax's own inability to hit zero by growing a handful of 'massive activations' — input-agnostic values up to 100,000× larger than the rest — that act as implicit bias terms, parking excess attention probability onto specific throwaway tokens Do hidden massive activations act as attention bias terms?. In other words, the architecture invents its own scratch space to dump attention it can't eliminate.
The consequences show up as behavior, not just internals. Models engage with conversational distractors and drift off-topic — and the striking finding is that this is a *training-signal* gap, not a capacity one: fine-tuning on just ~1,080 dialogues with distractor turns teaches models the 'what-to-ignore' instructions they never learned Why do language models engage with conversational distractors?. Relatedly, when a strong prior from pretraining competes with the actual context, the diffuse attention lets parametric knowledge override what's in front of the model, and textual prompting alone can't fix it Why do language models ignore information in their context?.
The interesting turn is that researchers are treating attention allocation as something to directly control rather than tolerate. One line makes the attention distribution itself the target of reinforcement learning, arguing that's where the real decision happens Can optimizing attention patterns improve multimodal RL better than optimizing tokens?. Another proves you can *train* sparsity in from scratch — a block selector learned during native pretraining matches full dense attention while cutting per-token compute 28× at million-token scale, showing focus doesn't have to cost quality sparse-attention-earns-its-place-by-surviving-native-pretraining-at-frontier-scale. And 'System 2 Attention' sidesteps the mechanism entirely by regenerating the context to strip irrelevant material before the model attends to it Does transformer attention architecture inherently favor repeated content?.
What you didn't come here knowing: the leak isn't a failure of the model to 'pay attention' — it's baked into the math of a function that can't say zero, and the field's most promising fixes either cancel the noise (differential attention), route around it (sparse selection, context regeneration), or make the distribution itself a first-class thing you optimize.
Sources 7 notes
DIFF Transformer computes attention as the difference between two softmax maps, mechanically canceling the shared noise floor while preserving differential signal—analogous to noise-cancelling headphones. This produces sparse, focused attention patterns without sacrificing efficiency.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Fine-tuning on just 1,080 synthetic dialogues with distractor turns significantly improves topic resilience, revealing that the gap is not model capacity but absent training signal. Models learn to follow what-to-do instructions but not what-to-ignore instructions.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Show all 6 sources
Reinforced Attention Learning treats attention patterns as the primary policy target rather than token sequences. Direct optimization of information allocation shows stronger gains on visual reasoning than standard RLHF, because attention is where the actual decision happens.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- System 2 Attention (is something you might need too)
- Differential Transformer
- It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
- Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
- TransformerFAM: Feedback attention is working memory
- Massive Activations in Large Language Models
- CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialogues
- Reinforced Attention Learning