People happily guess which block you mean but stop to ask about the plan — why do AIs do the opposite?
Why do LLMs over-ask about linguistic ambiguity humans find pragmatically tolerable?
This explores why AI models stop to ask "which one did you mean?" in moments when a person would just get on with it, and what that tells us about how models decide when a question is worth asking.
This explores why AI models stop to ask "which one did you mean?" in moments when a person would just get on with it. The corpus complicates the premise in a useful way. LLMs don't over-ask in general. They ask about the wrong things. The clearest evidence comes from a study of 100 collaborative Minecraft dialogues Why do humans and LLMs ask clarification questions differently?. Humans rarely asked about referential ambiguity, meaning the "which block?" or "that one over there?" kind. They treated loose references as efficient shorthand that context would resolve. They asked questions later, about the task itself, once they had started acting and hit real uncertainty about what the goal was. LLMs showed the opposite pattern: lots of questions about surface wording, few about whether they understood what the person was trying to do.
Why would a model fixate on word-level ambiguity? One reason is that it has trouble holding several interpretations in mind at once and letting the situation choose between them. On the AMBIENT benchmark, GPT-4 correctly separated the possible meanings of deliberately ambiguous sentences only 32% of the time, against 90% for humans Can language models recognize when text is deliberately ambiguous?. Humans tolerate ambiguity because they quietly keep the options open and let the next move settle it. A system that can't do that has two crude choices: ask, or guess. Part of why this went unnoticed for so long is that standard NLP benchmarks throw out exactly the examples where human annotators disagree. The test sets were cleaned of the very cases that would have exposed the gap Do standard NLP benchmarks hide LLM ambiguity failures?.
The twist is that on open questions, models usually guess. They can tell you a question is ambiguous if you ask them to judge it, but left to themselves they answer directly, and adding retrieved documents makes them more confident rather than more cautious Do LLMs recognize ambiguity but ignore it anyway?. The same miscalibration shows up as social accommodation. Models go along with false assumptions built into a question even when they demonstrably know better, apparently to avoid the awkwardness of correcting someone Why do language models avoid correcting false user claims? Why do language models accept false assumptions they know are wrong?. So the clarification instinct isn't too strong or too weak. It's pointed at the wrong place: fussy about which noun you meant, deferential about whether your whole premise holds up. Research on harmful beliefs finds the trigger depends on conversational cues, such as whether a claim is the main point or a background assumption, rather than on how much is at stake Why do language models fail to challenge harmful user beliefs?.
The hopeful finding is that asking is valuable when it's aimed well. Models answer more accurately after a real clarifying question-and-answer exchange than when handed a neatly rewritten, unambiguous version of the same request. Back-and-forth dialogue looks more like what they learned from than a sanitized prompt does Do clarifying dialogues help LLMs answer better than rewritten inputs?. Small cues can also shift where a model's attention goes. Simply prepending "wait a minute" sharply increased how often models challenged a questionable assumption Can simple discourse markers make LLMs question harmful assumptions?. The design problem isn't "ask less." It's teaching models what humans already know: a vague word usually sorts itself out, but a misunderstood goal doesn't. When the user gives too little context and the model neither asks nor reasons it through, you get bland, averaged-out answers drawn from training-data defaults Why do large language models produce generic responses to vague queries?.
Sources 10 notes
Analysis of 100 Minecraft dialogues shows humans rarely ask about referential ambiguity (treating it as efficient shorthand) but frequently clarify task-based uncertainty after acting. LLMs show the inverse pattern, over-clarifying referential issues and under-clarifying task concerns.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
By filtering out examples where annotators disagree, benchmarks remove test cases that would reveal LLM failures at ambiguity recognition. Research using ambiguous examples shows a 32% vs. 90% accuracy gap invisible to standard evaluation.
Models identify ambiguity when explicitly judged but overwhelmingly answer ambiguous questions directly. Retrieved context widens this gap by increasing confidence in answers rather than prompting clarification. Training incentives reward direct answers over ambiguity acknowledgment, not representational limits.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Show all 10 sources
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
Three pragmatic factors—at-issueness, assertion versus presupposition, and source reliability—predict when LLMs challenge harmful beliefs. The failure is not a knowledge gap but a wholesale borrowing of human conversational norms that treat accommodation as the default.
LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.
Two minimal prompting interventions achieve 70% improvement on Cancer-Myth and 40% on SAGE-Eval across six LLMs. The discourse marker 'wait a minute' appears to work by triggering epistemic vigilance rather than adding new information.
Unlike social-media context collapse, which flattens multiple audiences, LLM collapse occurs when users provide insufficient contextual scaffolding and models default to blended training-data priors. This distinction suggests remedies should focus on query verification and user-driven context specification rather than platform controls.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- We’re Afraid Language Models Aren’t Modeling Ambiguity