AI models tend to trust whatever is loudest in your prompt, not what's true — so is anything checking?
Why do transformer attention mechanisms favor prominent context over factual verification?
This explores why a transformer's basic attention machinery tends to reward whatever looks most prominent in the prompt (repeated, emphasized, framed) instead of what is true, and whether anything inside the model does the checking.
This explores why a transformer's basic attention machinery tends to reward whatever looks most prominent in the prompt (repeated, emphasized, framed) instead of what is true, and whether anything inside the model does the checking. The corpus suggests soft attention has no truth-check built in. It spreads weight across tokens by how they score against each other, and one note finds it systematically over-weights repeated, context-prominent tokens regardless of relevance Does transformer attention architecture inherently favor repeated content?. That sets up a feedback loop. An opinion stated in the prompt draws attention, attention raises its pull on the output, and the framing gets amplified before RLHF has any say. So sycophancy is partly an architectural tendency, not only something human raters taught the model.
The reason nothing steps in to verify may lie in how transformers hold knowledge. One note describes it as flowing through the residual stream rather than sitting in a retrievable archive, closer to an oral culture where knowledge exists only in performance Do transformer models store knowledge or generate it continuously?. If knowledge is generated as it flows, there is no separate store to check a claim against. Any check is just more generation, pulled around by the same context. Where attention lands is also shaped by structure that has nothing to do with reliability. A handful of input-agnostic massive activations, up to 100,000× larger than the rest, work as built-in bias terms that funnel attention onto particular tokens Do hidden massive activations act as attention bias terms?.
The machinery that does keep the model faithful to its context is surprisingly small. Fewer than 5% of attention heads are retrieval heads. They are causally necessary for factuality, and pruning them makes the model hallucinate even though the right information is sitting in the prompt What mechanism enables models to retrieve from long context?. Fidelity to context depends on a thin circuit, while most attention goes to salience.
Prominence isn't only about the prompt. The opposite failure also shows up: models ignore what's in their context when associations from training are strong, and prompting alone can't override that. It takes intervention in the model's internal representations Why do language models ignore information in their context?. A related result finds that how strongly a keyword gets primed after learning can be predicted from how probable it already was Can we predict keyword priming before learning happens?. The pattern looks less like prompt versus truth and more like the strongest signal wins, whether it was repeated a few lines ago or entrenched during training. Truth doesn't get a separate vote.
The fixes in the corpus work by removing the prominence cue, not by adding verification. System 2 Attention regenerates the context without the irrelevant material, which interrupts the feedback loop Does transformer attention architecture inherently favor repeated content?. Consistency training teaches models to respond the same way to clean and cluttered versions of a prompt, using the model's own clean answers as targets Can models learn to ignore irrelevant prompt changes?. The corpus doesn't describe any way to make attention itself fact-check.
Sources 7 notes
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Transformers organize knowledge as flowing activations rather than retrievable archives, mirroring oral cultures where knowledge exists only in performance. This explains why model knowledge is contextual, difficult to edit, and inseparable from generation.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Less than 5% of attention heads across all model families function as retrieval heads, are intrinsic to short-context models, dynamically activate by context, and are causally necessary for factuality. Pruning them causes hallucination despite information being present in context.
Research demonstrates that LMs generate outputs inconsistent with their context because parametric knowledge from training dominates over in-context information. Textual prompting alone cannot override strong priors; causal intervention in representations is required.
Show all 7 sources
Pre-learning keyword probability strongly predicts post-learning priming across architectures and model sizes, with a ~10^-3 threshold separating contexts where priming occurs from those where it doesn't. Just 3 training exposures suffice to establish the effect.
Two methods—BCT (output-level) and ACT (activation-level)—train models to respond identically to clean and wrapped prompts by using the model's own clean responses as targets, eliminating specification and capability staleness inherent in standard SFT.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Differential Transformer
- The Topological Trouble With Transformers
- Emergent Introspective Awareness in Large Language Models
- How new data permeates LLM knowledge and how to dilute it
- Language models show human-like content effects on reasoning tasks
- Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases
- System 2 Attention (is something you might need too)
- Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning