AI models can sense a sentence has multiple meanings, yet still can't spell out what those meanings actually are.
Why do LLMs fail to generate correct disambiguations despite recognizing ambiguity?
This explores why language models can notice that a sentence has more than one meaning but still can't reliably spell out what those meanings are, and it checks whether the premise holds, since the corpus suggests the recognition step is weaker than it looks.
This explores the gap between a model noticing that a sentence is ambiguous and actually producing the different readings it could have. The corpus partly pushes back on the question itself. On the AMBIENT benchmark, GPT-4 correctly disambiguated only 32% of deliberately ambiguous sentences, against 90% for humans. That failure covered word-level ambiguity, sentence-structure ambiguity and scope ambiguity (Can language models recognize when text is deliberately ambiguous?). The best explanation on offer is not that models know the answer and fumble the output. It's that they have trouble holding two interpretations open at the same time. A model can say "this is ambiguous" because that phrase is a familiar pattern, then fall back on the single most likely reading when asked to write out the alternatives.
The corpus has a name for this split between saying and doing: Potemkin understanding. Here a model explains a concept correctly, fails to apply it, and can even recognize that it failed. That pattern suggests explaining and executing run on separate tracks (Can LLMs understand concepts they cannot apply?). Ambiguity fits the pattern well. Labeling a sentence as ambiguous is the explanation. Producing both parses is the application. Related work on entailment shows the same habit: models treat cues like "believes" or "managed to" as surface signals instead of working out what those words do to meaning (Why do embedding contexts confuse LLM entailment predictions?). Errors also grow predictably as sentences nest more clauses (Why do large language models fail at complex linguistic tasks?). Structural ambiguity is exactly where you need that deeper parsing.
The frame-problem research suggests a fix. Models often have the relevant background knowledge but don't bring it forward unprompted. Forcing them to list unstated preconditions raised accuracy from 30% to 85% (Do language models fail at identifying unstated preconditions?). Disambiguation may be the same kind of enumeration failure. The second reading is available to the model, but nothing in ordinary generation asks it to surface that reading. That fits a related finding: when a user's query is too thin, models default to a blended, generic reading drawn from their training data instead of splitting the possibilities apart (Why do large language models produce generic responses to vague queries?).
The less obvious finding is about format, not ability. Models answered ambiguous questions more accurately when given a short clarifying back-and-forth than when given a cleanly rewritten, unambiguous version of the same question (Do clarifying dialogues help LLMs answer better than rewritten inputs?). Dialogue looks like the text they were trained on, while carefully edited rewrites don't. So models may be better at resolving ambiguity through conversation than at listing interpretations on demand.
This weakness also hides from most leaderboards. Standard NLP benchmarks routinely drop the examples where human annotators disagree, and those are exactly the ambiguous cases (Do standard NLP benchmarks hide LLM ambiguity failures?). The gap you're asking about stays invisible unless someone builds a test specifically to find it.
Sources 8 notes
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
Models can explain concepts accurately, fail to apply them, and recognize the failure—a triple pattern incompatible with human cognition. This indicates functionally disconnected explanation and execution pathways rather than simple knowledge gaps.
LLMs treat presupposition triggers and non-factive verbs as surface cues rather than computing their opposite semantic effects on entailments. This structural failure persists across prompts and models, suggesting models rely on surface patterns instead of structural analysis.
Top-tier LLMs like Llama3-70b consistently misidentify embedded clauses, verb phrases, and complex nominals. Performance degrades predictably as syntactic depth increases, revealing that statistical learning captures surface patterns but not deep grammatical rules.
LLMs struggle not from lacking world knowledge but from failing to bring background conditions forward as relevant constraints. Prompting that forces explicit enumeration of preconditions raises accuracy from 30% to 85%, revealing the frame problem persists in statistical systems.
Show all 8 sources
Unlike social-media context collapse, which flattens multiple audiences, LLM collapse occurs when users provide insufficient contextual scaffolding and models default to blended training-data priors. This distinction suggests remedies should focus on query verification and user-driven context specification rather than platform controls.
LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.
By filtering out examples where annotators disagree, benchmarks remove test cases that would reveal LLM failures at ambiguity recognition. Research using ambiguous examples shows a 32% vs. 90% accuracy gap invisible to standard evaluation.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
- Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds
- Large Language Model Reasoning Failures
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- Probing Structured Semantics Understanding and Generation of Language Models via Question Answering
- Word Meanings in Transformer Language Models
- We’re Afraid Language Models Aren’t Modeling Ambiguity