Can we understand LLM mechanisms with only representational analysis?
Explores whether mapping what information a model encodes is sufficient for mechanistic understanding, or whether causal verification is equally necessary to claim genuine mechanism.
The implementation-level argument in Levels of Analysis for LLMs is that representational analysis and causal analysis are partners, not alternatives. Representational analysis maps what information a model encodes — which features, circuits, attention heads carry which signals. Causal analysis tests whether the information that is encoded actually drives behavior — through interventions, ablations, activation patches. Either method alone produces an incomplete account: a representation that is encoded but causally inert is a curiosity, and a causal effect with no representational characterization is unexplained.
The synergy matters because both methods can fool you alone. Representational analysis can identify features that correlate with behavior without showing they cause it — a classic confound. Causal analysis can demonstrate that intervening on some component changes behavior without telling you what that component encodes — the lesion shows damage but not function. The combination — representational analysis locates candidates, causal analysis tests their functional role — is what produces mechanistic claims rather than descriptive ones.
This has methodological consequences for interpretability research. Studies that report only feature visualizations or only activation patches contribute, but they do not close the loop. The convergent evidence comes from pairs: locate a candidate feature representationally, then verify it causally; identify a causal component, then map its representation. The literature on attention circuits, induction heads, and feature dictionaries has been moving toward this pairing.
For LLM understanding specifically, this template explains why some claimed "mechanisms" have not held up. They were representational without causal verification (a feature that looked like task encoding but did not drive task behavior) or causal without representational characterization (an intervention that mattered but described nothing). The discipline imported from cognitive neuroscience is to demand both.
Inquiring lines that read this note 68
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How faithfully do LLMs reflect their actual reasoning in outputs and explanations?- What distinguishes LLM fabrication from genuine theoretical reasoning?
- What internal mechanisms explain LLM reasoning and representation limits?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- What distinguishes genuine cultural understanding from exploited surface-level elimination strategies?
- What's the difference between representing world facts and generating world mechanisms?
- Can mechanistic interpretability reveal how ideologies decompose into simpler features?
- How much do mechanistic interpretability findings reflect true reasoning architecture?
- Are detection and identification of injections truly separable in neural circuits?
- Can mechanistic interpretability explain explanation-execution disconnection?
- Does causal intervention alone explain how neural mechanisms implement representations?
- Can attractor dynamics compete with input-based probing for characterizing model knowledge?
- Can geometric structure in representations exist without supporting functional mechanisms?
- How does mechanistic interpretability complement learning mechanics in explaining deep learning?
- How do classical mechanics and statistical mechanics provide methodological templates for learning theory?
- What distinguishes a representational feature from a causally inert correlation?
- How do ablation studies reveal function without representational characterization?
- Can interventions on model components prove mechanism without explaining encoding?
- Can mechanistic interpretability tools decode the biases alignment training conceals?
- Can representation analysis methods detect complex features models compute with?
- How do mechanistic features compare to natural language for interpretability?
- How do mechanistic interpretability tools help distinguish truthfulness from honesty?
- Do causal rules enforce robustness that statistical patterns alone cannot maintain?
- What makes causal belief networks more auditable than prompted personas?
- Can causal models be extended to include non-causal cognition?
- Are traditional cognitive theories missing interaction effects between mechanisms?
- Can LLMs reason through semantics without understanding causal mechanisms?
- How does semantic association differ from mechanistic causal reasoning?
- How does vehicle causality differ from content causality in physical systems?
- Why do LLMs reason fluently about causality but lack causal rigor?
- How does causal structure avoid behaviorist limitations in LLM social simulation?
- Why does masking future experts guarantee causal validity without external verification?
- What architectural changes would help LLMs distinguish causal relationships from temporal sequences?
- What audit techniques best complement each other for detecting hidden model goals?
- What makes attractor-based probing better for third-party model auditing than alternatives?
- Can models distinguish between truthfulness and honesty mechanistically?
- How much introspective capability do safety mechanisms actively suppress in models?
- Can LLMs have minimal introspection through causal linkage to internal states?
- Can functional behavior alone capture what makes something a genuine belief?
- Can a perfect behavioral simulation constitute genuine understanding or experience?
- How do world models create indirect causal grounding without physical environment contact?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- Can LLM semantic representations exist without causally influencing their generation output?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- What prevents LLM representations from causally influencing generation outputs?
- Does next-state prediction alone build mechanistic world models or just sophisticated interpolation?
- How do world models decompose between representation of facts versus generative mechanisms?
- How do delayed effects complicate causal attribution in agent systems?
- What distinguishes mechanical generation failures from deliberate behavioral withholding?
Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can cognitive science methods unlock how LLMs actually work?
Does Marr's three-level framework—developed to understand biological minds—offer interpretability researchers the structured methodology they need to decode opaque language models?
same paper, the framework
-
Can we predict where language models will fail?
Does characterizing the abstract computational problem an LLM solves—as a probability machine over sequences—let us predict which tasks it will struggle with systematically, before running experiments?
same paper, computational level companion
-
Can indirect psychology tests reveal what LLMs conceal about bias?
Alignment training teaches LLMs to refuse direct questions about bias, but do implicit psychological methods like the IAT expose the underlying associations that remain encoded in their representations?
same paper, algorithmic level companion
-
Do language model reasoning drafts faithfully represent their actual computation?
If models externalize reasoning in thinking drafts before answering, does the draft accurately reflect their internal process? This matters for AI safety monitoring and error detection.
adjacent: dual-dimension methodology in CoT
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Levels of Analysis for Large Language Models
- Mechanistic Indicators of Understanding in Large Language Models
- AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders
- Computational structuralism: Toward a formal theory of meaning in the age of digital intelligence
- Language Models’ Hall of Mirrors Problem: Why AI Alignment Requires Peircean Semiosis
- LLM Reasoning Is Latent, Not the Chain of Thought
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning
Original note title
mechanistic understanding of LLMs requires both representational analysis and causal analysis — either alone is insufficient