Can language models understand without actually executing correctly?
Do LLMs truly comprehend problem-solving principles if they consistently fail to apply them? This explores whether the gap between articulate explanations and failed actions points to a fundamental architectural limitation.
LLMs display surface fluency yet systematically fail at tasks requiring symbolic reasoning, arithmetic accuracy, and logical consistency. The diagnosis: a persistent gap between comprehension and competence, rooted not in knowledge access but in computational execution.
The paper names this "computational split-brain syndrome" — instruction and action pathways are geometrically and functionally dissociated within the model. The model can articulate the correct principle for how to solve a problem, then fail to apply that principle in the next step. This is not forgetting, not hallucination, not knowledge deficit — it is a structural disconnect between knowing-how-to-describe and knowing-how-to-do.
The failure recurs across domains: mathematical operations, relational inferences, logical deductions. The consistency across domains suggests an architectural rather than domain-specific cause. LLMs function as powerful pattern completion engines but lack the scaffolding for principled, compositional reasoning — structure for executing what they can describe.
This provides a mechanistic name for Can LLMs understand concepts they cannot apply?. Potemkin understanding names the phenomenon; computational split-brain names the mechanism. The geometric separation between instruction representations and execution pathways explains why the model can generate correct explanations and incorrect applications simultaneously without detecting the inconsistency.
It also concretizes Why do language models fail to act on their own reasoning?. The 87% vs 64% gap is the quantitative signature of the split-brain: the instruction pathway (rationale generation) and the execution pathway (action selection) draw on overlapping but dissociated representations.
The paper further argues that mechanistic interpretability findings may reflect training-specific pattern coordination rather than universal computational principles — the internal structures we discover may be execution artifacts, not reasoning architecture.
Planning as the paradigmatic test case. The 8-puzzle study (On the Limits of Innate Planning in Large Language Models) isolates two specific deficits: (1) brittle internal state representations leading to frequent invalid moves, and (2) weak heuristic planning with models entering loops or selecting actions that don't reduce distance to the goal. Even with an external move validator providing only valid moves, none of the models solve any puzzles. The comprehension-competence split is stark: models can articulate puzzle-solving strategies but cannot maintain accurate state representations across sequential moves. Since Can large language models actually create executable plans?, the gap widens with task complexity: 87% correct rationales → 64% correct actions → 12% executable plans → 0% puzzle solutions with validator assistance.
Inquiring lines that read this note 110
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How aligned are models' verbal explanations with their actual internal reasoning?- When does knowledge activation fail across different model architectures?
- Can models identify what information they are missing in underspecified problems?
- How does the knowing-doing gap relate to Potemkin understanding?
- Can models distinguish between logical impossibility and their own execution limits?
- How does belief-behavior inconsistency relate to instruction execution splits?
- When does a model's lack of interpretability become a genuine epistemic problem?
- Where do LLMs succeed at generation but struggle with evaluation?
- Why do LLMs generate ideas that sound novel but fail during execution?
- Why do LLMs excel at generation but struggle with evaluation?
- What specific execution barriers do LLM ideas encounter most frequently?
- What distinguishes planning knowledge from an executable plan that works?
- How does the outer loop escape its own LLM's knowledge boundaries when discovering mechanisms?
- What mechanism causes LLMs to plateau on numerical optimization tasks?
- How should organizations redesign workflows if LLMs cannot solve optimization directly?
- What latent mechanisms do LLMs use when they cannot execute iterative methods?
- Why do LLMs strip applicability conditions during memory abstraction?
- What prevents monolithic LLMs from coordinating decomposition with execution?
- Can we systematically enumerate LLM failure modes from first principles?
- Can LLMs simultaneously reason and optimize their own modules?
- How should LLM abstraction tools be evaluated without manual labeling?
- What structural limits prevent scaffold-level fixes from reaching LLM-level reasoning corruption?
- What institutional knowledge do dedicated software makers possess that LLMs lack?
- How much of LLM reasoning failure stems from missing knowledge versus signal weighting?
- Why do LLM explanations feel authoritative even when alignment with the model fails?
- What explains the 87 percent to 12 percent cliff in plan executability?
- Why does LLM knowledge fail to influence their actual outputs?
- Can LLMs explain concepts correctly while failing to use them?
- What causes LLMs to ignore unstated constraints they know about?
- Where do LLMs fail as knowledge systems compared to humans?
- Why do LLMs fail when asked to use counter-commonsense rules explicitly?
- Why do LLMs struggle with negation and exception handling?
- Why can't LLMs reason from first principles or initial commitments?
- Why do LLMs explain evidence accurately while missing its implications?
- Can LLMs learn to ask clarifying questions instead of guessing?
- How can a model explain something correctly yet fail to apply it?
- Why do LLMs systematically fail at information management in social interaction?
- Why do LLMs choose incorrect edits despite understanding the task?
- How do knowing and doing diverge in LLM decision-making?
- Can surface-level correctness hide failures in structural learning by LLMs?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- How do LLM explanations diverge from actual internal reasoning?
- Why don't LLM explanations predict what models would actually do?
- How do LLM behavioral profiles differ across prompt registers like advice versus task execution?
- What barriers prevent experts from specifying concepts for LLM extraction?
- Why do people misinterpret or misuse LLM outputs in practice?
- Can LLMs give correct answers without making those answers understandable to users?
- Can LLMs faithfully apply decision rules when given complete information?
- Why do LLM explanations feel convincing even when the underlying plan is wrong?
- What contexts beyond question answering might show different LLM explanation effects?
- What makes a problem instance unfamiliar to a language model?
- Why does homework adherence remain low despite advances in language model capability?
- What architectural changes would let language models develop genuine functional competence?
- When and what should a model actually decide to delegate?
- Why might oversight focused only on language model outputs miss the agentic layer?
- Why do LLM agents make promises without executing them?
- Why do LLMs explain correct reasoning but then choose greedy actions?
- Why do language models fail at planning despite understanding strategies?
- Do LLMs fail exploration because of context integration or computational limitations?
- Why do LLMs understand efficient language but fail to produce it?
- Why do benchmark tests fail to detect LLM comprehension gaps?
- Why do LLMs struggle to translate natural language into logical formalizations?
- Why do language models fail at understanding ambiguous or complex requirements?
- Can we use LLM language without adopting LLM assumptions?
- Does the alignment frame mislead us about what LLM problems actually are?
- How faithful are natural language explanations from LLMs really?
- Can LLMs reliably audit other language models for errors?
- Why do monological explanations fail to transfer understanding compared to dialogical ones?
- How do dialogue acts and explanation moves interact to predict understanding success?
- At what complexity does LLM discourse failure become practically harmful?
- What cognitive capacities do LLMs actually lack that commentary assumes they have?
- Can LLMs participate meaningfully in discourse without consciousness or understanding?
- Can behavioral self-awareness in LLMs extend to recognizing their own contradictions?
- Why do LLMs understand therapy techniques but fail to execute them?
- Why do user studies of explanations fail to predict deployed effectiveness?
- How can humans evaluate explanations from systems they did not train?
- What distinguishes a logically sound solution from an understood one?
- What distinguishes genuine understanding from correct output without coherent principles?
- What makes some interpretive postures stick while others fail to form?
- What makes action-producing models fail in ways text models typically do not?
- What structural features enable agents to detect when understanding has broken down?
- What structural limits prevent LLMs from abstracting moral principles?
- How do spoken expert discussions shape what LLMs cannot learn?
- What internal mechanisms explain LLM reasoning and representation limits?
- Why can LLMs interpret formal logic better than they generate it?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Do LLMs lack architectural scaffolding for compositional reasoning?
- Can LLMs reason through semantics without understanding causal mechanisms?
- What concrete problems do LLMs solve at the computational level?
- Why do LLMs reason fluently about causality but lack causal rigor?
- Can high test performance mask a complete absence of understanding?
- Why does FunSearch claim interpretability without measuring human comprehension?
- How can correct explanations coexist with failed applications in AI?
- What happens when students encounter errors they cannot resolve through prompting alone?
- Are poor plans and poor error diagnosis rooted in the same weakness?
- Can reasoning models succeed at logic but fail at execution?
- Why do reasoning model failures stem from execution rather than reasoning?
Related concepts in this collection 3
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can LLMs understand concepts they cannot apply?
Explores whether large language models can correctly explain ideas while simultaneously failing to use them—and whether that combination reveals something fundamentally different from ordinary mistakes.
Potemkin understanding is the phenomenon; split-brain is the mechanism
-
Why do language models fail to act on their own reasoning?
LLMs produce correct explanations far more often than they produce correct actions. What causes this knowing-doing gap, and can training methods close it?
the quantitative signature of the comprehension-competence dissociation
-
Do language models actually use their encoded knowledge?
Probes can detect that LMs encode facts internally, but do those encoded facts causally influence what the model generates? This explores the gap between knowing and doing.
the encoding≠generation gap is the representational version of the same split
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Language Model Reasoning Failures
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Are Emergent Abilities in Large Language Models just In-Context Learning?
- The Debate Over Understanding in AI's Large Language Models
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
Original note title
comprehension without competence is a distinct LLM failure mode — instruction and execution pathways are dissociated