Could a simple geometric fingerprint tell you in advance which parts of a neural network are secretly running a readable algorithm?
Can mechanistic signatures like cosine clustering predict which heads are programmable?
This explores whether cheap internal fingerprints of an attention head — like how tightly its activation vectors cluster by cosine similarity — could tell you in advance which heads are simple enough to be swapped out for hand-written or synthesized code.
This explores whether a cheap structural signature of an attention head could forecast which heads are 'programmable' — meaning replaceable by an equivalent chunk of executable code rather than left as an inscrutable matrix. The most direct anchor in the corpus is the finding that program synthesis can recover executable code matching 99% of a head's behavior, and that substituting the best-fit 30–40% of heads with those programs preserves question-answering ability Can executable programs replace transformer attention heads?. That result establishes that programmability is real and unevenly distributed — some heads are essentially little algorithms, others aren't — which is exactly the precondition for wanting a predictor. But notice the method used to find them: it fits an actual program and measures behavioral match. It does not screen heads by a geometric fingerprint first.
That gap matters, because the corpus is unusually skeptical that a representational signature alone can tell you what a component is doing. Mechanistic claims require both representational analysis (finding structure) and causal analysis (verifying it drives behavior); representation alone yields correlations, and causal probing alone yields effects without explanation Can we understand LLM mechanisms with only representational analysis?. Cosine clustering is a purely representational signal — so on this view it could nominate candidate heads, but couldn't confirm one is programmable without actually attempting the substitution and testing that behavior survives.
The sharpest warning comes from work showing that internal organization and measured competence can come apart entirely: a model can carry all the linearly decodable features a task needs while its internal structure is fundamentally fractured, a mismatch invisible to standard metrics Can models be smart without organized internal structure?. Flip that around and it cuts against clustering as a predictor twice over — a head whose vectors look tidily clustered may be doing something the clean geometry misrepresents, and a head that looks geometrically messy may still implement a crisp, extractable rule. Neat cosine structure is neither necessary nor sufficient for programmability.
There's also a deeper reason to expect the signature to be leaky. Transformers lack native recurrent state-tracking and smear evolving computation across layers as a workaround Why do transformers need explicit chain-of-thought reasoning?. A head's local activation geometry is only one slice of a computation that may be distributed through depth — so a head that reads as 'simple' by its own vectors could be one stage of something no single program captures. Programmability may be a property of a head's role in the circuit, not of its isolated activation cloud.
The honest read: the corpus supports the idea that heads differ in programmability and that this is worth predicting, but it does not contain a result validating cosine clustering — or any single representational signature — as the predictor. What it repeatedly says is that a representational signal is at best a candidate-generator that must be closed off with a causal test: try the substitution, and check the model still works Can executable programs replace transformer attention heads? Can we understand LLM mechanisms with only representational analysis?.
Sources 4 notes
Program synthesis recovers executable code matching 99% of attention head behavior. Substituting the best-fit 30–40% of heads with their synthesized programs preserves QA ability, offering formal, testable interpretability instead of natural-language summaries.
Representational analysis alone identifies correlations without causation; causal analysis alone shows behavioral effects without explaining them. Only paired methods—locating candidate features representationally, then verifying causally—produce complete mechanistic claims.
Models trained with SGD can contain all the linearly decodable features needed for a task while maintaining fundamentally broken internal organization. This makes them vulnerable to perturbation and distribution shift invisible to standard evaluation metrics.
Feedforward transformers lack native recurrent state-tracking and must push evolving state deeper into layers, eventually exhausting depth. Explicit chain-of-thought externalizes this state into tokens as a costly patch for a structural deficiency.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Explaining Attention with Program Synthesis
- The Topological Trouble With Transformers
- A Mechanistic Analysis of Looped Reasoning Language Models
- Levels of Analysis for Large Language Models
- Implicit Chain of Thought Reasoning via Knowledge Distillation