A third of transformer attention heads can be replaced with simple programs — but the other two-thirds resist, and that's the interesting part.
Why do some attention heads resist program synthesis better than others?
This explores why program synthesis can cleanly replace some transformer attention heads with executable code but chokes on others — and what that residual, un-synthesizable behavior reveals about what those heads are actually doing.
This explores why program synthesis can cleanly replace some transformer attention heads with executable code but chokes on others. The starting point is the striking result that you can synthesize small executable programs matching about 99% of a head's behavior, and swap out the best-fit 30–40% of heads with those programs while keeping question-answering intact Can executable programs replace transformer attention heads?. The unstated flip side is the more interesting story: 60–70% of heads resist. The question is what those hold-outs are computing that a compact program can't capture.
The corpus points to a few culprits. Some heads aren't doing token-to-token logic at all — they're carrying input-agnostic structural machinery. A tiny number of 'massive activations,' with values up to 100,000× the rest, act as implicit attention-bias terms that dump probability onto specific tokens regardless of content Do hidden massive activations act as attention bias terms?. A head whose behavior is dominated by that kind of always-on bias is a poor fit for a program that expects to read inputs and produce outputs; its 'function' is a constant, a plumbing role, not an algorithm.
Other heads resist because their behavior is contextual and self-referential in ways a static program can't mirror. Soft attention is structurally biased to over-weight whatever content is repeated or prominent, creating feedback loops that shift with the surrounding context rather than following a fixed rule Does transformer attention architecture inherently favor repeated content?. Related work on 'attention noise' shows that heads spread probability onto irrelevant context, and that this diffuse, unstable behavior is a shared root cause of hallucination and order-sensitive in-context learning Can fixing attention noise solve multiple LLM failures at once?. A head whose output is a noisy, distribution-dependent smear is exactly the kind of thing that fits well 'on average' but never reaches the crisp determinism a synthesized program needs — the 1% gap is where the interesting cognition lives.
The deepest resistance may come from heads doing genuinely dynamic, memory-like work. Feedback-attention architectures show transformers can attend to their own latent representations to build emergent working memory over unbounded sequences Can models learn working memory by attending to their own latents?. A head participating in that kind of recurrent, state-carrying loop isn't a pure function of the current input — its output depends on an evolving internal state, which a stateless synthesized program has no way to reconstruct.
So the synthesizable/un-synthesizable split doubles as a rough map of function: the replaceable heads tend to be doing local, rule-like copying and matching; the resistant ones are carrying bias, riding context-dependent feedback, or holding state. That reframes the whole exercise — the failure of program synthesis isn't a limitation of the method so much as a probe. The heads that resist are telling you where a transformer stops behaving like a lookup table and starts behaving like something with dynamics, which is why formal, testable program-substitution is a sharper interpretability tool than natural-language summaries that gloss every head as if it did the same kind of thing Can executable programs replace transformer attention heads?.
Sources 5 notes
Program synthesis recovers executable code matching 99% of attention head behavior. Substituting the best-fit 30–40% of heads with their synthesized programs preserves QA ability, offering formal, testable interpretability instead of natural-language summaries.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
Transformer soft attention systematically over-weights repeated and context-prominent tokens regardless of relevance, creating a positive feedback loop that amplifies opinions and framing before RLHF acts. System 2 Attention—regenerating context to remove irrelevant material—can interrupt this mechanism.
Cancelling attention noise through differential attention improves hallucination rates, makes in-context learning robust to example order, and reduces activation outliers—suggesting these separate failure modes share one root cause: over-attention to irrelevant context.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- TransformerFAM: Feedback attention is working memory
- System 2 Attention (is something you might need too)
- The Topological Trouble With Transformers
- Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference
- It’s All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
- Emergent Introspective Awareness in Large Language Models
- Massive Activations in Large Language Models
- Explaining Attention with Program Synthesis