SYNTHESIS NOTE
Can LLMs understand concepts they cannot apply?
Explores whether large language models can correctly explain ideas while simultaneously failing to use them—and whether that combination reveals something fundamentally different from ordinary mistakes.
The Potemkin understanding paper identifies a failure pattern that is categorically different from ordinary LLM error. When a model correctly explains an ABAB rhyme scheme, then fails to generate one, then recognizes that its generation doesn't rhyme — that triple combination is not just wrong, it is incoherent. No human with that explanation would behave that way. The combination is irreconcilable with any human cognitive pattern.
This is worth separating from other LLM failure types because the mechanism matters for diagnosis and repair:
- Ordinary errors (fabrication, factual mistakes) — the model lacks information or generates plausible-but-false continuations. Fix: better retrieval, grounding, training data.
- Surface generalizations — the model learned correlations that worked in training but don't generalize structurally. Fix: better training curriculum, structural probing.
- Potemkin understanding — the model can produce the explanation and fails to apply it and recognizes the failure. This combination implies that explanation-generation and concept-application are functionally disconnected. No single epistemic fix addresses both.
The "Potemkin" framing (after Potemkin villages — facades with nothing behind) is precise: the model passes benchmark tests designed to detect understanding because those benchmarks test the same cognitive operations as humans. The tests only work as diagnostics if LLMs misunderstand concepts the same way humans do. But Potemkin understanding means the model can perform at the surface without the underlying integration that tests were designed to probe.
Benchmarks used to evaluate LLMs are also used to evaluate people. They are valid tests only if LLMs fail in human-compatible ways. Potemkin understanding shows that this assumption fails — LLMs can fail in ways that no human cognitive model predicts.
The three-domain evidence (literary techniques, game theory, psychological biases) shows this is not domain-specific. Across domains: near-perfect explanation accuracy, significant application failure, model recognition of failure. The incoherence is stable.
The "computational split-brain syndrome" diagnosis. "Comprehension Without Competence" provides the architectural analysis underlying Potemkin understanding. Through controlled experiments, the authors demonstrate that instruction and action pathways are geometrically and functionally dissociated — a phenomenon they term computational split-brain syndrome. The failure is not in knowledge access but in computational execution. LLMs function as powerful pattern completion engines but lack the architectural scaffolding for principled, compositional reasoning. This diagnosis also clarifies why mechanistic interpretability findings may reflect training-specific pattern coordination rather than universal computational principles. The geometric separation between instruction and execution pathways represents a structural limitation, not a knowledge limitation.
The Explain-Query-Test (EQT) framework provides direct empirical measurement of the explanation-comprehension gap. In EQT, a model (1) generates an explanation of a topic, (2) generates question-answer pairs from that explanation, and (3) answers those same questions without access to its own explanation. The finding: models consistently fail questions derived from their own explanations. The EQT gap correlates strongly with MMLU-PRO benchmark performance — making EQT a benchmark-free evaluation method that uses only the model's own outputs as ground truth. Critically, the gap is domain-specific: biology and psychology (domains where models initially perform well) show the largest EQT drops, while law and engineering (lower baseline) show smaller drops. This suggests Potemkin understanding is worst precisely where surface performance is highest — a counterintuitive result that demands explanation. High benchmark performance may mask explanation-comprehension disconnection rather than reveal genuine understanding.
Inquiring lines that read this note 223
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Do language models genuinely understand or primarily match statistical patterns?- Why do some LLM clusters cite broader psychology than others?
- Can LLMs learn to signal evaluative commitment through metadiscursive language?
- What structural limits prevent LLMs from abstracting moral principles?
- Can LLMs identify implicit metaphoric mappings that require pragmatic inference?
- Do LLMs rely on surface statistical patterns instead of causal structure?
- What implicit knowledge about catalogs do LLMs learn from ranking signals alone?
- Why do LLMs rely on content knowledge instead of collaborative signals?
- Does LLM vocabulary become the cultural lexicon for how we think?
- Can LLMs infer situational context the way humans do pragmatically?
- How do fixed pragmatic templates prevent models from understanding context?
- Is interpretive multiplicity a bug in language or a feature?
- Why do LLMs produce semantically acceptable but pragmatically disengaged responses?
- Can explicit connectives compensate for missing intentional tracking in LLMs?
- Why do large language models fail at temporal reasoning in complex legal cases?
- How does implicit meaning processing limit LLM pragmatic reasoning?
- Can language models accurately evaluate the quality of their own ideas?
- Can training LLMs to form ad-hoc conventions improve their pragmatic reasoning?
- Do metaphors work by decoupling meaning from linguistic associations?
- Can language models distinguish between novel insight and unjustified conceptual blending?
- Can LLMs reliably audit other language models for errors?
- How do model compression biases differ from human conceptual representation strategies?
- Why do untrained LLMs default to rigid and sycophantic editing rules?
- How does content-only knowledge in LLMs enable pretraining popularity to leak through?
- Why does monological training prevent models from overriding statistical priors?
- What happens when LLMs analyze literary irony that relies on understatement?
- Why do LLM personas struggle with specificity in specialized domains like law?
- How much of LLM reasoning failure stems from missing knowledge versus signal weighting?
- Why do LLM explanations feel authoritative even when alignment with the model fails?
- Why does LLM knowledge fail to influence their actual outputs?
- Can LLMs explain concepts correctly while failing to use them?
- What causes LLMs to ignore unstated constraints they know about?
- What cognitive capacities do LLMs actually lack that commentary assumes they have?
- Why does semantic decoupling specifically break LLM reasoning abilities?
- Can LLMs infer implicit meaning without surface linguistic markers?
- Can LLMs distinguish ethical cases that differ only in critical nouns?
- Does exposure to more domain-specific examples reduce LLM overconfidence?
- Where do LLMs fail as knowledge systems compared to humans?
- Why do LLMs generate novel ideas but lack evaluative commitment?
- Why can LLMs identify argument structure but not check warrants?
- Why do LLMs fail when asked to use counter-commonsense rules explicitly?
- Why do LLMs struggle with negation and exception handling?
- Why can't LLMs reason from first principles or initial commitments?
- Why do LLMs explain evidence accurately while missing its implications?
- Why can LLMs interpret formal logic better than they generate it?
- Can training procedures fix LLM accommodation of false presuppositions?
- How much does question framing affect LLM accuracy on knowledge tasks?
- Can LLMs improve at simple deduction through different training approaches?
- How do LLMs handle false presuppositions embedded in user questions?
- How can a model explain something correctly yet fail to apply it?
- How does the LLM Fallacy prevent users from noticing cognitive debt accumulating?
- Why do experts experiencing the LLM Fallacy fail to develop custodian skills?
- Why do LLMs understand therapy techniques but fail to execute them?
- What training data barriers prevent LLMs from learning real Socratic dialogue?
- Is the distinction between pretense and realization meaningful for LLMs?
- Why do LLMs fail at counterfactual reasoning despite factual knowledge?
- Why do LLMs explain correct reasoning but then choose greedy actions?
- How do knowing and doing diverge in LLM decision-making?
- Can surface-level correctness hide failures in structural learning by LLMs?
- Why do LLM stories over-explain themes and favor single-track plots?
- Can irrelevant information reliably expose the limits of LLM reasoning?
- What structural framework prevents LLM explanations from becoming just plausible fiction?
- How do LLM explanations diverge from actual internal reasoning?
- Why do LLMs reason fluently about causality but lack causal rigor?
- Why does LLM fluency create false perceptions of professional standing and expertise?
- Why don't LLM explanations predict what models would actually do?
- How do prescriptive ethical constraints differ from descriptive ethical understanding in LLMs?
- Why do people misinterpret or misuse LLM outputs in practice?
- Can LLMs give correct answers without making those answers understandable to users?
- How persistent is LLM overconfidence in specialized knowledge domains?
- Why do LLM explanations feel convincing even when the underlying plan is wrong?
- Why do LLMs reason semantically rather than symbolically?
- What contexts beyond question answering might show different LLM explanation effects?
- Why do LLMs fail inter-annotator agreement tests on argument evaluation?
- What role do model-based critics play in validating LLM plans?
- When does an LLM judge's error become terrain that an optimizer can map and exploit?
- Why do LLMs swing on minor rewording yet ignore explicit bias correction instructions?
- Do LLMs lack evaluative capacity or only default taste and stance?
- How does LLM hallucination risk manifest in knowledge graph construction?
- Where do LLMs succeed at generation but struggle with evaluation?
- Why do LLMs generate ideas that sound novel but fail during execution?
- Why do LLM outputs match researcher priors without solving tasks correctly?
- Why do language models fail when semantic content is stripped away?
- How do rare linguistic registers differ from conceptually complex examples?
- Can LLMs improve at metaphor if they handle decoupled semantics better?
- Why do LLMs excel at generation but struggle with evaluation?
- Why do large language models still have systematic blind spots with complex structures?
- Can knowledge density explain why LLM writing feels coherent but fatiguing?
- Do LLMs struggle more with semantic accuracy than syntactic correctness across domains?
- Why do LLMs fail at implicit elements in literary and poetic text?
- Can complexity-stratified testing reveal whether LLMs understand grammatical structure?
- Why do LLMs fail at semantic generalization despite grammatical accuracy?
- What distinguishes entity errors from relation errors in LLM output?
- Why do NLP benchmarks hide LLM failures in ambiguity handling?
- Can LLMs translate between natural language and formal logic faithfully?
- Do standard language benchmarks underestimate what LLMs can actually do?
- Do LLMs generate more novel ideas than they can evaluate?
- Why do standard NLP benchmarks hide the most critical language limitations?
- How does the inability to manage ambiguity undermine literary analysis tasks?
- Can LLM semantic representations exist without causally influencing their generation output?
- How much semantic meaning survives when LLMs paraphrase poetry and literary text?
- Why do LLMs understand efficient language but fail to produce it?
- How does structural complexity in sentences degrade LLM reasoning systematically?
- Can LLMs compute how presuppositions project through embedded clauses?
- Can LLMs recognize rhetorical devices they cannot actually produce themselves?
- Can LLMs distinguish stylistic patterns that carry meaning from mere convention?
- Why do LLMs generate novel ideas but struggle to evaluate them?
- What structural barriers prevent LLMs from making evaluative judgments about writing?
- Why do LLMs struggle to translate natural language into logical formalizations?
- Can LLMs generate more novel research ideas than human experts?
- Why do backward-looking benchmarks underestimate LLM scientific value?
- Can LLMs reason through semantics without understanding causal mechanisms?
- Does compressing Walton's schemes into nine categories make LLM classification easier?
- What concrete problems do LLMs solve at the computational level?
- Why do LLM descriptions of argument schemes work better than formal definitions for classification?
- Why do language models fail at understanding ambiguous or complex requirements?
- Can we use LLM language without adopting LLM assumptions?
- How do LLMs lose information when translating natural language to formal logic?
- Why do LLMs fail at faithful autoformalisation of reasoning problems?
- How does the pretraining distribution shape what LLMs find hard?
- How faithful are natural language explanations from LLMs really?
- Can language models translate theorems faithfully without semantic loss?
- What barriers prevent experts from specifying concepts for LLM extraction?
- Does LLM miscalibration cause failures in clinical information extraction?
- How do spoken expert discussions shape what LLMs cannot learn?
- Why do LLMs miss new scientific ideas before they enter formal literature?
- How do LLMs build concepts without embodied experience or mental models?
- What specific execution barriers do LLM ideas encounter most frequently?
- Can output-layer corrections fix fundamental cultural representation deficits in LLMs?
- How does the outer loop escape its own LLM's knowledge boundaries when discovering mechanisms?
- Why do language models fail at planning despite understanding strategies?
- How do LLMs compress specific expert knowledge into median abstraction?
- Can pruning half of LLM layers affect knowledge retrieval performance?
- What internal mechanisms explain LLM reasoning and representation limits?
- Do LLMs fail exploration because of context integration or computational limitations?
- How does structural complexity affect LLM performance differently than inferential complexity?
- Can LLMs reliably generate novel working architectures without structured representations?
- Why do LLMs recognize graph entities without modeling their relationships?
- What latent mechanisms do LLMs use when they cannot execute iterative methods?
- Why do LLMs choose incorrect edits despite understanding the task?
- Why do LLMs strip applicability conditions during memory abstraction?
- Why do LLMs struggle more when only numerical values change?
- What capability boundary exists in LLM prediction of effect sizes?
- Why does over-specialization create a domain capability cliff in LLMs?
- Do LLMs detect harmful concepts before they influence model outputs?
- How should LLM abstraction tools be evaluated without manual labeling?
- What levels of understanding about LLM knowledge representation can automated systems reliably extract?
- Why do LLMs excel at isolated tasks but fail at integrating components across long texts?
- What makes a problem instance unfamiliar to a language model?
- Is relevant knowledge encoded in LMs but not causally active in generation?
- What reveals the epistemic limits of language models?
- Can encoder models match human conceptual structure better than larger language models?
- Why do NLP models fail at recognizing multiple valid interpretations?
- Can linear probing detect all the concepts a language model actually uses?
- How widespread is task contamination in LLM evaluation benchmarks today?
- Why do NLP benchmarks systematically exclude ambiguous test cases from evaluation?
- Why do NLP benchmarks exclude ambiguous instances from evaluation?
- Why does homework adherence remain low despite advances in language model capability?
- Why do benchmark tests fail to detect LLM comprehension gaps?
- Can models identify what information they are missing in underspecified problems?
- Can LLMs learn to ask clarifying questions instead of guessing?
- Why do text-to-image models fail at composing multiple concepts together?
- Why do multimodal models fail on rare and underrepresented concepts?
- Why do reasoning models fail on structurally unfamiliar instances?
- Why do models overthink underspecified problems instead of rejecting them?
- Why do reasoning models fail at learning hidden rules from sparse exceptions?
- Why does the Chinese Room argument miss the deeper abstraction problem?
- How does instance novelty rather than chain length explain reasoning failure?
- How do dependency errors propagate through incorrectly formalized definitions?
- Can problem structure and representation format be mismatched intentionally?
- Why do monological explanations fail to transfer understanding compared to dialogical ones?
- At what complexity does LLM discourse failure become practically harmful?
- What linguistic blind spots do LLMs exhibit in discourse structure?
- What happens when formal languages satisfy hierarchy but fail learnability constraints?
- Which linguistic abilities are learnable from human-sized data exposure?
- Can instance seeds work for tasks beyond language understanding benchmarks?
- What distinguishes surface generalizations from true linguistic generalizations?
- Why do surface generalizations fail on unusual syntactic structures?
- Do LLMs learn linguistic generalizations or just surface-level frequency patterns?
- Do LLMs learn surface patterns instead of genuine linguistic structure?
- Why does entity recognition act as a self-knowledge mechanism in LLMs?
- Can behavioral self-awareness in LLMs extend to recognizing their own contradictions?
- How can we probe LLM representations in channels that training did not target?
- Can LLMs decode their own hidden activations into natural language?
- Can representation engineering reliably identify and manipulate self-referential concepts in models?
- Why does AI struggle with wordplay when it has access to word embeddings?
- How can humans evaluate explanations from systems they did not train?
- Why does FunSearch claim interpretability without measuring human comprehension?
- What happens when technological capacity outpaces ordinary language comprehension?
- How do we distinguish knowledge encoding from knowledge usage in models?
- What distinguishes conceptual understanding from statistical pattern matching in models?
- What prevents LLM representations from causally influencing generation outputs?
- When does a model's lack of interpretability become a genuine epistemic problem?
- Does more thinking always help large language models or sometimes hurt?
- Why does chain-of-thought reasoning alone not fix LLM performance with users?
- What role does failure and vulnerability play in real linguistic practice?
- Can grammar alone repair misunderstanding without ritual correction work?
- Why do rare complex structures in training data harm LLM generalization?
- What data presentation structures enable LLMs to learn decision-making from examples?
- Can understanding language happen entirely within a language system alone?
- How does the knowing-doing gap relate to Potemkin understanding?
- How do single-ordering encyclopedic systems limit ways of knowing?
- How might human-LLM teams reinforce each other's causal reasoning mistakes?
- Why does LLM compression eliminate causal grounding in conceptual representations?
- How should we rethink the symbolism versus connectionism debate in light of LLMs?
- How do concept vectors in neural networks relate to human-language chess explanations?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Concept map
Can LLMs understand concepts they cannot apply?
Do language models actually use their encoded know…
Can models pass tests while missing the actual gra…
Do language models actually use their reasoning st…
Can identical outputs hide broken internal represe…
What limits how much models can improve themselves…
Why do language models fail to act on their own re…
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
your link
semantically near
linked from elsewhere
-
Do language models actually use their encoded knowledge?
Probes can detect that LMs encode facts internally, but do those encoded facts causally influence what the model generates? This explores the gap between knowing and doing.
related mechanism: knowledge can be present without causally influencing behavior; Potemkin extends this to a more observable test (explanation vs. application)
-
Can models pass tests while missing the actual grammar?
Do language models succeed on grammatical benchmarks by learning surface patterns rather than structural rules? This matters because correct outputs may hide reliance on shallow heuristics that fail on novel structures.
Potemkin understanding adds the recognition-of-failure component that surface generalization accounts don't predict
-
Do language models actually use their reasoning steps?
Chain-of-thought reasoning looks valid on the surface, but does each step genuinely influence the model's final answer, or are the reasoning chains decorative? This matters for trusting AI explanations.
faithful reasoning would prevent Potemkin: the explanation would causally constrain the application
-
Can identical outputs hide broken internal representations?
Can neural networks produce correct outputs while having fundamentally fractured internal structure that prevents generalization and creativity? This challenges our assumptions about what performance benchmarks actually measure.
FER provides the mechanistic cause of Potemkin understanding: the internal representation is fractured across arbitrary subdomains and entangled across unrelated computations, which is why explanation-generation and concept-application are functionally disconnected despite identical surface performance
-
What limits how much models can improve themselves?
Explores whether self-improvement has fundamental boundaries set by how well models can verify versus generate solutions, and what this means across different task types.
the generation-verification gap formalizes why Potemkin understanding paradoxically enables self-improvement: when explanation exceeds application, that gap is a usable training signal — the model's verification ability can supervise its generation ability
-
Why do language models fail to act on their own reasoning?
LLMs produce correct explanations far more often than they produce correct actions. What causes this knowing-doing gap, and can training methods close it?
quantified instance: the 87%/64% gap between correct rationales and correct actions in sequential decision-making is the most precisely measured example of Potemkin understanding; RL fine-tuning narrows the gap, suggesting the facade is partially trainable
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Large Language Model Reasoning Failures
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- The Debate Over Understanding in AI's Large Language Models
- Word Meanings in Transformer Language Models
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
Original note title
potemkin understanding is a distinct failure mode where correct explanation combined with failed application is incoherent not merely wrong