SYNTHESIS NOTE
Can LLMs understand concepts they cannot apply?
Explores whether large language models can correctly explain ideas while simultaneously failing to use them—and whether that combination reveals something fundamentally different from ordinary mistakes.
The Potemkin understanding paper identifies a failure pattern that is categorically different from ordinary LLM error. When a model correctly explains an ABAB rhyme scheme, then fails to generate one, then recognizes that its generation doesn't rhyme — that triple combination is not just wrong, it is incoherent. No human with that explanation would behave that way. The combination is irreconcilable with any human cognitive pattern.
This is worth separating from other LLM failure types because the mechanism matters for diagnosis and repair:
- Ordinary errors (fabrication, factual mistakes) — the model lacks information or generates plausible-but-false continuations. Fix: better retrieval, grounding, training data.
- Surface generalizations — the model learned correlations that worked in training but don't generalize structurally. Fix: better training curriculum, structural probing.
- Potemkin understanding — the model can produce the explanation and fails to apply it and recognizes the failure. This combination implies that explanation-generation and concept-application are functionally disconnected. No single epistemic fix addresses both.
The "Potemkin" framing (after Potemkin villages — facades with nothing behind) is precise: the model passes benchmark tests designed to detect understanding because those benchmarks test the same cognitive operations as humans. The tests only work as diagnostics if LLMs misunderstand concepts the same way humans do. But Potemkin understanding means the model can perform at the surface without the underlying integration that tests were designed to probe.
Benchmarks used to evaluate LLMs are also used to evaluate people. They are valid tests only if LLMs fail in human-compatible ways. Potemkin understanding shows that this assumption fails — LLMs can fail in ways that no human cognitive model predicts.
The three-domain evidence (literary techniques, game theory, psychological biases) shows this is not domain-specific. Across domains: near-perfect explanation accuracy, significant application failure, model recognition of failure. The incoherence is stable.
The "computational split-brain syndrome" diagnosis. "Comprehension Without Competence" provides the architectural analysis underlying Potemkin understanding. Through controlled experiments, the authors demonstrate that instruction and action pathways are geometrically and functionally dissociated — a phenomenon they term computational split-brain syndrome. The failure is not in knowledge access but in computational execution. LLMs function as powerful pattern completion engines but lack the architectural scaffolding for principled, compositional reasoning. This diagnosis also clarifies why mechanistic interpretability findings may reflect training-specific pattern coordination rather than universal computational principles. The geometric separation between instruction and execution pathways represents a structural limitation, not a knowledge limitation.
The Explain-Query-Test (EQT) framework provides direct empirical measurement of the explanation-comprehension gap. In EQT, a model (1) generates an explanation of a topic, (2) generates question-answer pairs from that explanation, and (3) answers those same questions without access to its own explanation. The finding: models consistently fail questions derived from their own explanations. The EQT gap correlates strongly with MMLU-PRO benchmark performance — making EQT a benchmark-free evaluation method that uses only the model's own outputs as ground truth. Critically, the gap is domain-specific: biology and psychology (domains where models initially perform well) show the largest EQT drops, while law and engineering (lower baseline) show smaller drops. This suggests Potemkin understanding is worst precisely where surface performance is highest — a counterintuitive result that demands explanation. High benchmark performance may mask explanation-comprehension disconnection rather than reveal genuine understanding.
Inquiring lines that read this note 189
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do evaluation biases undermine LLM quality assessment systems?- Why do some LLM clusters cite broader psychology than others?
- Why do LLM outputs match researcher priors without solving tasks correctly?
- Can knowledge density explain why LLM writing feels coherent but fatiguing?
- Can LLMs learn to signal evaluative commitment through metadiscursive language?
- Does exposure to more domain-specific examples reduce LLM overconfidence?
- Can language models accurately evaluate the quality of their own ideas?
- Why do backward-looking benchmarks underestimate LLM scientific value?
- Why does LLM fluency create false perceptions of professional standing and expertise?
- What capability boundary exists in LLM prediction of effect sizes?
- Can LLMs infer situational context the way humans do pragmatically?
- Can LLMs use implicit background knowledge the way humans do in ordinary conversation?
- What cognitive capacities do LLMs actually lack that commentary assumes they have?
- How does implicit meaning processing limit LLM pragmatic reasoning?
- Can LLMs infer implicit meaning without surface linguistic markers?
- What structural limits prevent LLMs from abstracting moral principles?
- Can training LLMs to form ad-hoc conventions improve their pragmatic reasoning?
- Can LLMs identify implicit metaphoric mappings that require pragmatic inference?
- How does content-only knowledge in LLMs enable pretraining popularity to leak through?
- How does the outer loop escape its own LLM's knowledge boundaries when discovering mechanisms?
- What implicit knowledge about catalogs do LLMs learn from ranking signals alone?
- What should we call errors in LLM outputs when hallucination does not apply?
- How does LLM hallucination risk manifest in knowledge graph construction?
- Can output-layer corrections fix fundamental cultural representation deficits in LLMs?
- What distinguishes entity errors from relation errors in LLM output?
- Where do LLMs fail as knowledge systems compared to humans?
- How does the LLM Fallacy prevent users from noticing cognitive debt accumulating?
- Why do experts experiencing the LLM Fallacy fail to develop custodian skills?
- Why do LLMs recognize graph entities without modeling their relationships?
- Can surface-level correctness hide failures in structural learning by LLMs?
- Do LLMs detect harmful concepts before they influence model outputs?
- How do fixed pragmatic templates prevent models from understanding context?
- Can explicit connectives compensate for missing intentional tracking in LLMs?
- Can LLMs compute how presuppositions project through embedded clauses?
- What linguistic blind spots do LLMs exhibit in discourse structure?
- What happens when LLMs analyze literary irony that relies on understatement?
- Why do LLM personas struggle with specificity in specialized domains like law?
- How much of LLM reasoning failure stems from missing knowledge versus signal weighting?
- Why do LLM explanations feel authoritative even when alignment with the model fails?
- Why does LLM knowledge fail to influence their actual outputs?
- Can LLMs explain concepts correctly while failing to use them?
- What causes LLMs to ignore unstated constraints they know about?
- Can LLMs distinguish ethical cases that differ only in critical nouns?
- What internal mechanisms explain LLM reasoning and representation limits?
- Why can LLMs identify argument structure but not check warrants?
- Why do LLMs fail when asked to use counter-commonsense rules explicitly?
- Why do LLMs struggle with negation and exception handling?
- Why can't LLMs reason from first principles or initial commitments?
- Why do LLMs explain evidence accurately while missing its implications?
- Can training procedures fix LLM accommodation of false presuppositions?
- How much does question framing affect LLM accuracy on knowledge tasks?
- Can LLMs improve at simple deduction through different training approaches?
- How can a model explain something correctly yet fail to apply it?
- What training data barriers prevent LLMs from learning real Socratic dialogue?
- How do structured benchmarks hide theory of mind failures in LLMs?
- Why do LLMs fail at counterfactual reasoning despite factual knowledge?
- Does compressing Walton's schemes into nine categories make LLM classification easier?
- Why do LLMs explain correct reasoning but then choose greedy actions?
- Why do LLMs choose incorrect edits despite understanding the task?
- How do knowing and doing diverge in LLM decision-making?
- Why do LLM stories over-explain themes and favor single-track plots?
- Can irrelevant information reliably expose the limits of LLM reasoning?
- How faithful are natural language explanations from LLMs really?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- How do LLM explanations diverge from actual internal reasoning?
- Why don't LLM explanations predict what models would actually do?
- Why do LLMs fail inter-annotator agreement tests on argument evaluation?
- What happens when LLMs grade other LLMs in closed evaluation loops?
- Where do LLMs succeed at generation but struggle with evaluation?
- Why do LLMs excel at generation but struggle with evaluation?
- Why do LLMs generate novel ideas but lack evaluative commitment?
- Do LLMs generate more novel ideas than they can evaluate?
- Can LLMs recognize rhetorical devices they cannot actually produce themselves?
- Why do LLMs generate novel ideas but struggle to evaluate them?
- What structural barriers prevent LLMs from making evaluative judgments about writing?
- What role do model-based critics play in validating LLM plans?
- Why do LLMs generate ideas that sound novel but fail during execution?
- What specific execution barriers do LLM ideas encounter most frequently?
- Why does LLM research ideation collapse into low diversity despite high novelty?
- What makes a novel research idea practically infeasible for implementation?
- Can LLMs generate more novel research ideas than human experts?
- What makes a problem instance unfamiliar to a language model?
- Can instance seeds work for tasks beyond language understanding benchmarks?
- How does the pretraining distribution shape what LLMs find hard?
- How widespread is task contamination in LLM evaluation benchmarks today?
- Why does homework adherence remain low despite advances in language model capability?
- Can models identify what information they are missing in underspecified problems?
- Can LLMs learn to ask clarifying questions instead of guessing?
- Why do NLP benchmarks systematically exclude ambiguous test cases from evaluation?
- Why do NLP benchmarks exclude ambiguous instances from evaluation?
- Why do language models fail at planning despite understanding strategies?
- Why do large language models still have systematic blind spots with complex structures?
- Why do rare complex structures in training data harm LLM generalization?
- Why do NLP benchmarks hide LLM failures in ambiguity handling?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Why do standard NLP benchmarks hide the most critical language limitations?
- Do LLMs fail exploration because of context integration or computational limitations?
- Why do benchmark tests fail to detect LLM comprehension gaps?
- What latent mechanisms do LLMs use when they cannot execute iterative methods?
- Why do language models fail at understanding ambiguous or complex requirements?
- Why do LLMs struggle more when only numerical values change?
- Is interpretive multiplicity a bug in language or a feature?
- Why do language models fail when semantic content is stripped away?
- How do rare linguistic registers differ from conceptually complex examples?
- Can LLMs improve at metaphor if they handle decoupled semantics better?
- Do LLMs struggle more with semantic accuracy than syntactic correctness across domains?
- Why do LLMs fail at implicit elements in literary and poetic text?
- Why do LLMs fail at semantic generalization despite grammatical accuracy?
- Do metaphors work by decoupling meaning from linguistic associations?
- How does the inability to manage ambiguity undermine literary analysis tasks?
- How much semantic meaning survives when LLMs paraphrase poetry and literary text?
- Why do NLP models fail at recognizing multiple valid interpretations?
- Can large language models actually deliver cognitive behavioral therapy techniques?
- Why do LLMs understand therapy techniques but fail to execute them?
- Why do reasoning models fail on structurally unfamiliar instances?
- Does more thinking always help large language models or sometimes hurt?
- Why do models overthink underspecified problems instead of rejecting them?
- Why do reasoning models fail at learning hidden rules from sparse exceptions?
- Why does the Chinese Room argument miss the deeper abstraction problem?
- How does instance novelty rather than chain length explain reasoning failure?
- How do dependency errors propagate through incorrectly formalized definitions?
- Why do monological explanations fail to transfer understanding compared to dialogical ones?
- Can grammar alone repair misunderstanding without ritual correction work?
- Why do LLMs produce semantically acceptable but pragmatically disengaged responses?
- How do LLMs handle false presuppositions embedded in user questions?
- Can language models distinguish between novel insight and unjustified conceptual blending?
- Can LLMs reliably audit other language models for errors?
- Why do large language models fail at temporal reasoning in complex legal cases?
- What happens when formal languages satisfy hierarchy but fail learnability constraints?
- Which linguistic abilities are learnable from human-sized data exposure?
- Is relevant knowledge encoded in LMs but not causally active in generation?
- What reveals the epistemic limits of language models?
- Can complexity-stratified testing reveal whether LLMs understand grammatical structure?
- Can encoder models match human conceptual structure better than larger language models?
- What distinguishes surface generalizations from true linguistic generalizations?
- Why do surface generalizations fail on unusual syntactic structures?
- Do LLMs learn linguistic generalizations or just surface-level frequency patterns?
- Why do LLMs understand efficient language but fail to produce it?
- Do LLMs learn surface patterns instead of genuine linguistic structure?
- Can linear probing detect all the concepts a language model actually uses?
- How do model compression biases differ from human conceptual representation strategies?
- Why does semantic decoupling specifically break LLM reasoning abilities?
- Can LLMs translate between natural language and formal logic faithfully?
- Why can LLMs interpret formal logic better than they generate it?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Can LLMs reliably generate novel working architectures without structured representations?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- Why do LLMs struggle to translate natural language into logical formalizations?
- What concrete problems do LLMs solve at the computational level?
- Why do LLM descriptions of argument schemes work better than formal definitions for classification?
- Can we use LLM language without adopting LLM assumptions?
- How do LLMs lose information when translating natural language to formal logic?
- Why do LLMs fail at faithful autoformalisation of reasoning problems?
- Can language models translate theorems faithfully without semantic loss?
- How do LLMs compress specific expert knowledge into median abstraction?
- How do we distinguish knowledge encoding from knowledge usage in models?
- Can pruning half of LLM layers affect knowledge retrieval performance?
- What distinguishes conceptual understanding from statistical pattern matching in models?
- Why does entity recognition act as a self-knowledge mechanism in LLMs?
- Can behavioral self-awareness in LLMs extend to recognizing their own contradictions?
- What role does failure and vulnerability play in real linguistic practice?
- Can understanding language happen entirely within a language system alone?
- How might human-LLM teams reinforce each other's causal reasoning mistakes?
- Why does LLM compression eliminate causal grounding in conceptual representations?
- Can LLMs reason through semantics without understanding causal mechanisms?
- How does the knowing-doing gap relate to Potemkin understanding?
- Why do LLMs reason fluently about causality but lack causal rigor?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- Can LLM semantic representations exist without causally influencing their generation output?
- What data presentation structures enable LLMs to learn decision-making from examples?
- How can we probe LLM representations in channels that training did not target?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- What prevents LLM representations from causally influencing generation outputs?
- What happens when students encounter errors they cannot resolve through prompting alone?
- Do rare cultural concepts fail predictably as model scale increases?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can LLMs understand concepts they cannot apply?
Do language models actually use their encoded know…
Can models pass tests while missing the actual gra…
Do language models actually use their reasoning st…
Can identical outputs hide broken internal represe…
What limits how much models can improve themselves…
Why do language models fail to act on their own re…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do language models actually use their encoded knowledge?
Probes can detect that LMs encode facts internally, but do those encoded facts causally influence what the model generates? This explores the gap between knowing and doing.
related mechanism: knowledge can be present without causally influencing behavior; Potemkin extends this to a more observable test (explanation vs. application)
-
Can models pass tests while missing the actual grammar?
Do language models succeed on grammatical benchmarks by learning surface patterns rather than structural rules? This matters because correct outputs may hide reliance on shallow heuristics that fail on novel structures.
Potemkin understanding adds the recognition-of-failure component that surface generalization accounts don't predict
-
Do language models actually use their reasoning steps?
Chain-of-thought reasoning looks valid on the surface, but does each step genuinely influence the model's final answer, or are the reasoning chains decorative? This matters for trusting AI explanations.
faithful reasoning would prevent Potemkin: the explanation would causally constrain the application
-
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
FER provides the mechanistic cause of Potemkin understanding: the internal representation is fractured across arbitrary subdomains and entangled across unrelated computations, which is why explanation-generation and concept-application are functionally disconnected despite identical surface performance
-
What limits how much models can improve themselves?
Explores whether self-improvement has fundamental boundaries set by how well models can verify versus generate solutions, and what this means across different task types.
the generation-verification gap formalizes why Potemkin understanding paradoxically enables self-improvement: when explanation exceeds application, that gap is a usable training signal — the model's verification ability can supervise its generation ability
-
Why do language models fail to act on their own reasoning?
LLMs produce correct explanations far more often than they produce correct actions. What causes this knowing-doing gap, and can training methods close it?
quantified instance: the 87%/64% gap between correct rationales and correct actions in sequential decision-making is the most precisely measured example of Potemkin understanding; RL fine-tuning narrows the gap, suggesting the facade is partially trainable
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Large Language Model Reasoning Failures
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Word Meanings in Transformer Language Models
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
Original note title
potemkin understanding is a distinct failure mode where correct explanation combined with failed application is incoherent not merely wrong