Can executable programs replace transformer attention heads?

Most interpretability labels network components with human-defined concepts (probing) or natural-language summaries — but those descriptions are themselves ambiguous and resist formalization. "Explaining Attention with Program Synthesis" takes a stricter target: to explain what an attention head computes, search for a piece of executable code that reproduces it. The explanation is not a paraphrase; it is a program you can run and check.

The results give interpretability a falsifiable currency. Synthesized programs achieve up to 99% mean IoU similarity against observed attention patterns, and there is a clean negative correlation between a program's fit and the perplexity increase caused by replacing the head with that program — better-fit programs are safer to substitute. Crucially, up to 30–40% of all attention heads can be replaced by their highest-similarity programs without losing downstream QA ability, which means for a substantial fraction of heads the program almost fully captures the head's functional role. The honest limit: many heads still score below 40% IoU, and the QA degradation at aggressive replacement traces to those poorly-fit heads — no model is yet fully characterized.

This raises the evidentiary bar for the vault's interpretability thread. A natural-language claim like "this head does induction" is unfalsifiable in the way a program isn't — the program either reproduces the activation and survives ablation-substitution, or it doesn't. It also gestures at a compression story: since Can a single transformer become universally programmable through prompts?, transformer heads are executing programs in a real sense, and this work is beginning to recover those programs head-by-head. The unrecovered ≤40%-IoU tail marks exactly where transformer computation is still genuinely non-program-like.