Most of an AI's attention can be turned into plain code — what are the other 60% actually doing?
What computation remains in the attention heads that programs cannot capture?
This explores what transformer attention heads actually compute once you strip away the parts that can be mechanically rewritten as code — the residue that resists formal capture.
This explores what's left inside attention heads after you subtract the behavior that can be turned into plain executable code. The starting point is a striking result: program synthesis can recover code matching 99% of a given attention head's behavior, and swapping the best-fit 30–40% of heads for those programs preserves a model's question-answering ability Can executable programs replace transformer attention heads?. Read carefully, that number cuts both ways — it means a majority of heads (the other 60–70%) do *not* cleanly reduce to a program. The interesting question isn't the part that compiles; it's the part that doesn't.
One clue about the irreducible remainder is that some heads do a job that's less an algorithm and more a dynamic, content-triggered routing. Fewer than 5% of heads act as 'retrieval heads' — a sparse, universal mechanism that activates depending on what's in the context and is causally necessary for pulling facts out of long inputs; prune them and the model hallucinates even though the answer sits right there in the window What mechanism enables models to retrieve from long context?. A program that fits one context's activation pattern misses the point: the head's value is that it *decides when to fire* across contexts, not that it runs a fixed procedure. Similar story with the handful of input-agnostic 'massive activations' that behave like implicit attention bias terms, silently shaping where probability mass lands Do hidden massive activations act as attention bias terms?. These are global, model-wide regularities, not per-head logic you can transcribe.
There's also a messier residue: noise that turns out to be functional. Cancelling 'attention noise' simultaneously reduces hallucination, stabilizes in-context learning against example ordering, and shrinks activation outliers — suggesting these look like separate bugs but share one root in over-attention to irrelevant context Can fixing attention noise solve multiple LLM failures at once?. A program-synthesis pass that faithfully copies a head would copy this noise too, because from the outside it's indistinguishable from signal. The head's real computation is entangled with its failure modes in a way that clean code abstracts away.
The deepest gap may be temporal. A frozen program captures a head as a snapshot, but a growing line of work treats attention's job as maintaining and updating *state* over time — the long-context bottleneck is now framed not as memory capacity but as the compute needed to consolidate evicted context into fast weights Is long-context bottleneck really about memory or compute?, and architectures like TransformerFAM add feedback loops so a model attends to its own latents and grows an emergent working memory Can models learn working memory by attending to their own latents?. Neural-memory designs like Titans go further, splitting short-term attention from a long-term module that decides which surprising tokens are worth keeping Can neural memory modules scale language models beyond attention limits?. Programs capture the *when-you-give-me-X-I-return-Y* face of a head. What they miss is the head as a participant in an ongoing, stateful, self-modifying process.
So the honest answer is layered: what programs can't capture is (1) content-conditional *routing* — the decision of when to activate, not the fixed transform once activated; (2) global, cross-head regularities like bias-term massive activations that no single head 'owns'; (3) functional noise that's inseparable from signal at the behavioral surface; and (4) the temporal, state-carrying role attention plays across a sequence. There's even a further wrinkle worth sitting with: work decomposing chain-of-thought finds that a lot of apparent 'reasoning' is really output probability and memorization, with genuine step-by-step reasoning a smaller, error-accumulating third factor What three separate factors drive chain-of-thought performance? — which raises the possibility that the truly program-resistant computation isn't some deep algorithm at all, but a statistical shaping of probability that only looks like computation from the outside.
Sources 8 notes
Program synthesis recovers executable code matching 99% of attention head behavior. Substituting the best-fit 30–40% of heads with their synthesized programs preserves QA ability, offering formal, testable interpretability instead of natural-language summaries.
Less than 5% of attention heads across all model families function as retrieval heads, are intrinsic to short-context models, dynamically activate by context, and are causally necessary for factuality. Pruning them causes hallucination despite information being present in context.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Cancelling attention noise through differential attention improves hallucination rates, makes in-context learning robust to example order, and reduces activation outliers—suggesting these separate failure modes share one root cause: over-attention to irrelevant context.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Show all 8 sources
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
A shift cipher study decomposed CoT into three independent factors: output probability alone swings accuracy from 26% to 70%, memorization matches pre-training frequency patterns, and genuine reasoning exists but accumulates error with each step. This resolves the reason-or-memorize debate by showing LLMs do both simultaneously.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
- Titans: Learning to Memorize at Test Time
- In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- Language Models Need Sleep
- Explaining Attention with Program Synthesis
- The Topological Trouble With Transformers
- Emergent Introspective Awareness in Large Language Models