How does attention sink behavior relate to internal model architecture?
This explores attention sinks — the way transformers dump attention onto a few special tokens — and what that reveals about how the architecture is actually built, not just how it behaves.
This explores attention sinks — the well-known habit of transformers parking large amounts of attention probability on a handful of tokens (often the first one) — and what that tells us about the model's internal wiring rather than its visible outputs. The corpus's sharpest answer is that the "sink" isn't a quirk of the input; it's structural. A tiny number of input-agnostic "massive activations" — values up to 100,000× larger than their neighbors — act as implicit bias terms baked into the network, and they're what concentrate attention onto specific tokens Do hidden massive activations act as attention bias terms?. Because they show up across model sizes and even in Vision Transformers, the sink looks less like a learned response to particular text and more like a load-bearing feature the architecture needs to function.
Why would attention need somewhere to dump itself? Part of the answer is that softmax attention is structurally biased to begin with. It systematically over-weights repeated and context-prominent tokens regardless of whether they're relevant, creating feedback loops that amplify whatever is already prominent Does transformer attention architecture inherently favor repeated content?. A model that must distribute attention probability that always sums to one needs an outlet when no token is genuinely worth attending to — and a fixed, input-agnostic sink token is a tidy place to send the leftover mass. The two findings dovetail: the bias terms create the sink, and the sink relieves the pressure that softmax's structural over-weighting would otherwise put on real content.
The deeper, more uncomfortable lesson comes from mechanistic interpretability: a model's internal structure and its external performance are decoupled. Networks can hit identical accuracy while running radically different internal representations, and mechanisms that look interpretable may not actually drive the output What actually happens inside the minds of language models?. So attention sinks are a case study in why you can't read architecture off behavior — the sink is visible in the attention map, but its real role lives in those hidden activation magnitudes, not in anything the model "says." This is the same gap that shows up when researchers find that reasoning traces are persuasive appearances rather than records of computation Do reasoning traces show how models actually think?, or that model self-reports mostly echo training data rather than genuine introspection Can language models actually introspect about their own states?.
If the sink is a workaround for what attention structurally can't do, one response is to stop asking attention to do all the work. The Titans architecture splits short-term attention from a separate neural memory module that adaptively stores surprising tokens, scaling past two million tokens without the quadratic cost — and without leaning on attention as the only mechanism for holding information Can neural memory modules scale language models beyond attention limits?. Read alongside the massive-activations finding, this suggests attention sinks aren't a bug to patch but a symptom of asking one mechanism to handle both "what to focus on" and "where to put the overflow." The interesting frontier isn't eliminating sinks — it's recognizing them as evidence that the architecture is quietly improvising structure the design didn't explicitly give it.
Sources 6 notes
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
LLMs can achieve identical accuracy while maintaining radically different internal representations, and mechanisms that appear interpretable may not causally drive outputs. This decoupling means performance metrics alone mask crucial differences in how models actually work.
LLM reasoning traces perform as persuasive appearances rather than reliable explanations of computation. Invalid logical steps perform nearly as well as valid ones, and corrupted traces generalize comparably, showing that semantic correctness is not what produces the performance gains.
LLM self-reports usually reflect human training distributions rather than actual internal processes. However, when a causal chain connects an internal state to accurate reporting—like inferring low temperature from output consistency—genuine lightweight introspection occurs without requiring consciousness.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.