A transformer gives extra weight to anything said twice — so the AI's tendency to agree with you is partly just arithmetic.

This explores the mechanism inside the transformer itself — how attention over-weights repeated and prominent content as a structural property, not a learned quirk. The clearest account is that soft attention systematically assigns extra weight to tokens that recur or sit prominently in context, regardless of whether they're actually relevant. Because attention is a soft weighted average, a token that appears twice pulls more probability mass than one that appears once, and that creates a positive feedback loop: opinions, framing, and repeated assertions get amplified before RLHF ever touches the model. This is why sycophancy is partly baked into the architecture rather than purely a training artifact — the fix proposed, System 2 Attention, works by regenerating the context to strip irrelevant material so the averaging has less junk to over-weight Does transformer attention architecture inherently favor repeated content?.

Underneath that, there's an even more literal source of bias. A tiny handful of activations — input-agnostic, with values up to 100,000× larger than everything around them — act as implicit bias terms that concentrate attention probability onto particular tokens. These aren't semantic; they show up across model sizes and even in vision transformers, meaning the attention distribution has hard-wired attractors independent of what the input actually says Do hidden massive activations act as attention bias terms?. So 'prominence' is partly a property of the model's internal machinery, not just the text in front of it.

The repetition bias also compounds with how learning works. Whether a keyword gets primed after a gradient update is predictable from its probability *before* learning — above a roughly 10^-3 threshold priming reliably kicks in, and just three exposures are enough to lock it in Can we predict keyword priming before learning happens?. That's the same shape as the attention effect at a different timescale: things that are already prominent get preferentially reinforced, whether across a context window or across training. Repetition doesn't just get echoed, it gets entrenched.

One reframing worth carrying away: this bias is inseparable from how transformers hold knowledge at all. Rather than storing facts as retrievable records, they transmit knowledge as continuous flow through the residual stream — knowledge exists only in the act of generation, closer to oral performance than to an archive Do transformer models store knowledge or generate it continuously?. If knowledge is a live performance shaped by whatever is loudest in context, then over-weighting the prominent isn't a bug to patch — it's the same mechanism that lets the model speak at all. That's why these biases are so hard to edit out cleanly.

The corpus also points at the escape routes, which tell you what the default bias is by contrast. Architectures like Titans add a separate neural memory that deliberately prioritizes *surprising* tokens rather than prominent ones, pushing against attention's over-weighting of the familiar Can neural memory modules scale language models beyond attention limits?. And consistency training teaches a model to respond identically whether or not irrelevant framing is wrapped around a prompt, using the model's own clean answers as targets Can models learn to ignore irrelevant prompt changes?. Both are admissions that, left alone, attention will follow the loud and the repeated — and that counteracting it takes an explicit second mechanism.