Why do people wait to ask questions about the goal, while AI rushes to ask which object you mean?
Why do humans and LLMs ask clarification questions differently?
This explores why people and LLMs ask for clarification about different things and at different moments, and what that difference shows about how each one handles shared understanding in conversation.
This explores why people and language models ask for clarification about different things and at different times. The corpus suggests the two have opposite instincts. In a study of 100 collaborative Minecraft building dialogues, humans rarely asked which block or which spot the speaker meant. They treated vague references as efficient shorthand and worked them out from context. Their questions came later, about the task itself, often after they had already started acting and run into uncertainty. LLMs did the reverse: they over-asked about which thing was meant and under-asked about whether they understood the goal Why do humans and LLMs ask clarification questions differently?. People clarify to keep the work on track. Models clarify the words.
The gap doesn't seem to come from models failing to notice ambiguity. Asked directly whether a question is ambiguous, models usually say yes, and then answer it anyway. Giving them retrieved documents makes this worse, because the extra context makes them more confident rather than more cautious. The likely cause is training that rewards direct answers, not an inability to see the problem Do LLMs recognize ambiguity but ignore it anyway?. The same pattern appears when a question contains a false assumption. Models often go along with the premise even when they demonstrably know it is wrong Why do language models accept false assumptions they know are wrong?. Answering is the default, and checking in has to work against it.
Looking at conversation more broadly, clarification questions are one part of what linguists call grounding. These are the small moves people use to confirm they're on the same page, like acknowledging, repairing a misunderstanding or checking in. LLMs make these moves 77.5% less often than humans. They assume shared understanding instead of building it, and fluent, authoritative wording hides the gap Do language models actually build shared understanding in conversation?. There's also a measurement problem: most human clarifications aren't phrased as questions at all. They're statements like "the red one, I think" or "wait, you mean the left side." They also happen at different levels, from "did you hear me?" to "what do you want me to do?" Why do clarification requests look different at each communication level?. A system or benchmark that only counts sentences ending in a question mark misses most of what people actually do.
The partial fixes are informative. Reasoning models and chain-of-thought prompting make clarification questions both more frequent and more relevant. One reading is that thinking step by step gives the model room to imagine what its partner knows and doesn't know Does reasoning ability help LLMs ask better clarification questions?. And when clarification does happen, it pays off more than you might expect. Models answer more accurately when given a real clarifying exchange than when given a neatly rewritten, unambiguous version of the same question, apparently because dialogue looks more like what they learned from Do clarifying dialogues help LLMs answer better than rewritten inputs?.
The reason this matters beyond tidiness: when a model skips the check-in, the reader usually can't tell. Fluent explanations raise trust in wrong answers just as much as in right ones Do LLM explanations actually help people judge answer quality?, and default explanations leave people more confident than the model's own internal probabilities justify Why do readers trust LLM explanations more than they should?. A model that asks about the wrong things therefore doesn't just waste a turn. It can quietly replace a misunderstanding you would have caught with one you won't.
Sources 9 notes
Analysis of 100 Minecraft dialogues shows humans rarely ask about referential ambiguity (treating it as efficient shorthand) but frequently clarify task-based uncertainty after acting. LLMs show the inverse pattern, over-clarifying referential issues and under-clarifying task concerns.
Models identify ambiguity when explicitly judged but overwhelmingly answer ambiguous questions directly. Retrieved context widens this gap by increasing confidence in answers rather than prompting clarification. Training incentives reward direct answers over ambiguity acknowledgment, not representational limits.
The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.
LLMs produce grounding acts—clarifications, acknowledgments, repairs—77.5% less frequently than humans. They generate fluent responses without verifying shared understanding, relying instead on authoritative framing that masks the absence of genuine communicative calibration.
Research maps clarification mechanisms to four levels of communication—attention, signal, meaning, action—each grounded in a different modality (socioperception, hearing, vision, kinesthetics). Most clarifications use declarative form, not questions, making them invisible to systems that detect by syntax alone.
Show all 9 sources
Testing reasoning-tuned models and chain-of-thought prompting against baseline LLMs on a Minecraft dialogue task, researchers found reasoning increased both the frequency and relevance of clarification questions. The link suggests reasoning may support metacognitive reflection and simulation of others' knowledge.
LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.
A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.
Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Referential ambiguity and clarification requests: comparing human and LLM behaviour
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Fostering Appropriate Reliance on Large Language Models: The Role of Explanations, Sources, and Inconsistencies