Even when an expert can explain a concept perfectly, an AI may still fail to apply it. So how do you spell it out?
What barriers prevent experts from specifying concepts for LLM extraction?
This explores what gets in the way when a domain expert tries to define a concept (a category, a label, a distinction) precisely enough for an LLM to pull matching examples out of text.
This explores what gets in the way when a domain expert tries to define a concept precisely enough for an LLM to pull matching examples out of text. The corpus has no note on the expert's side of the problem, such as tacit knowledge that resists being written down. What it does have is a set of documented model-side failures that would make any written specification unreliable, and they suggest why experts struggle.
The first barrier is that a successful-looking specification can fail. Models can explain a concept correctly, fail to apply it, and even recognize the failure afterward. The corpus calls this Can LLMs understand concepts they cannot apply?. One study measured 87% accuracy in explanations against 64% in actions, which points to Can language models understand without actually executing correctly?: the instruction pathway and the execution pathway seem to be disconnected. An expert who checks a spec by asking the model to restate the concept gets false reassurance. The restatement can be perfect while the extraction is wrong. The broader catalogue in How do LLMs fail to know what they seem to understand? shows this is one repeatable pattern among several.
The second barrier is ambiguity, which experts resolve without noticing. GPT-4 correctly disambiguates only 32% of ambiguous cases, against 90% for humans, according to Can language models recognize when text is deliberately ambiguous?. The model can't hold several readings at once. That note is about ambiguity in text being read, not in instructions, so applying it to specifications is my inference. A specification written in plain language carries the same gaps that a human colleague fills from shared context, and the model may pick one reading silently instead of asking which one you meant.
The third barrier is that an expert's meaning may get flattened. Can language models distinguish expert arguments from common assumptions? says models can't tell an expert's argument from a commonly held assumption, because they see only text and not the standing behind it. It's plausible that a term with a narrow technical sense gets read in its everyday sense, though the note doesn't test this. Do language models understand in fundamentally different ways? adds that models can represent a concept well and still coexist with cruder shortcuts. So a model can appear to follow the expert's definition while extracting by surface similarity.
The fourth barrier is verification. Refining a spec depends on seeing where it went wrong. Does model capability change how documents degrade? finds that weaker models fail visibly by deleting content, while frontier models corrupt it in ways that preserve surface integrity. The better the model, the harder the expert's feedback loop becomes, because errors look plausible.
So the corpus can't tell you why experts find concepts hard to articulate. It does suggest that even a well-articulated concept has no reliable route from the expert's head to the model's behavior, and that checking whether it arrived gets harder as models improve.
Sources 7 notes
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
Large language models can articulate correct principles but systematically fail to apply them due to dissociated instruction and execution pathways. The 87% accuracy in explanations versus 64% in actions reveals this is not knowledge deficit but structural disconnect.
LLMs show repeatable, empirically documented failure modes—from Potemkin understanding (correct explanation + failed application) to reasoning collapse under implicit constraints. These failures reveal gaps between statistical pattern-tracking and actual epistemic competence.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
LLMs lose the social context that gives expert claims their force—reputation, track record, and standing—because they process only text, not the social world where expertise is built and evaluated.
Show all 7 sources
Mechanistic interpretability reveals conceptual understanding (features as directions), state-of-world understanding (factual connections), and principled understanding (compact circuits). Crucially, higher tiers coexist with lower-tier heuristics rather than replacing them, creating a patchwork of capabilities.
DELEGATE-52 shows weaker LLMs degrade documents through visible deletion, while frontier models degrade through subtle corruption that preserves surface integrity. This shift makes frontier failures harder to detect and potentially more dangerous at workflow scale.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Large Language Model Reasoning Failures
- Explain-Query-Test: Self-Evaluating LLMs Via Explanation and Comprehension Discrepancy
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Comprehension Without Competence: Architectural Limits of LLMs in Symbolic Computation and Reasoning
- Word Meanings in Transformer Language Models
- Six misconceptions about large language models: A minimal model and diagnostic taxonomy
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens