Can executable programs replace transformer attention heads?
Can we write code that mimics what attention heads do, and if so, how many heads could we swap out without breaking task performance? This tests whether interpretability claims can be made falsifiable and formal.
Most interpretability labels network components with human-defined concepts (probing) or natural-language summaries — but those descriptions are themselves ambiguous and resist formalization. "Explaining Attention with Program Synthesis" takes a stricter target: to explain what an attention head computes, search for a piece of executable code that reproduces it. The explanation is not a paraphrase; it is a program you can run and check.
The results give interpretability a falsifiable currency. Synthesized programs achieve up to 99% mean IoU similarity against observed attention patterns, and there is a clean negative correlation between a program's fit and the perplexity increase caused by replacing the head with that program — better-fit programs are safer to substitute. Crucially, up to 30–40% of all attention heads can be replaced by their highest-similarity programs without losing downstream QA ability, which means for a substantial fraction of heads the program almost fully captures the head's functional role. The honest limit: many heads still score below 40% IoU, and the QA degradation at aggressive replacement traces to those poorly-fit heads — no model is yet fully characterized.
This raises the evidentiary bar for the vault's interpretability thread. A natural-language claim like "this head does induction" is unfalsifiable in the way a program isn't — the program either reproduces the activation and survives ablation-substitution, or it doesn't. It also gestures at a compression story: since Can a single transformer become universally programmable through prompts?, transformer heads are executing programs in a real sense, and this work is beginning to recover those programs head-by-head. The unrecovered ≤40%-IoU tail marks exactly where transformer computation is still genuinely non-program-like.
Inquiring lines that read this note 14
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do transformer circuits and attention mechanisms enable or limit reasoning?- Why do some attention heads resist program synthesis better than others?
- Can mechanistic signatures like cosine clustering predict which heads are programmable?
- How does program synthesis relate to transformers computing general algorithms?
- What computation remains in the attention heads that programs cannot capture?
- How do retrieval heads achieve sparse attention naturally in transformers?
- How do attention circuits demonstrate both representational and causal findings?
- How does this attention mechanism differ from RLHF-driven intent alignment failures?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can a single transformer become universally programmable through prompts?
Explores whether prompts can function as genuine programs that unlock universal computation in fixed-size models, and whether this theoretical possibility translates to practical training outcomes.
relates: if heads compute programs, program synthesis recovers them
-
How do transformers learn to reason across multiple steps?
Does multi-hop reasoning in transformers emerge through distinct learning phases, and what geometric patterns in hidden representations explain when reasoning succeeds or fails?
relates: complementary mechanistic account of what heads implement
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Explaining Attention with Program Synthesis
- Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- Measuring and Controlling Instruction (In)Stability in Language Model Dialogs
- Emergent Introspective Awareness in Large Language Models
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity
- Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought
Original note title
attention heads can be replaced by synthesized executable programs — up to 40 percent swapped out without losing task ability