Humans ask fewer clarifying questions than AI, but somehow ask about the right things more often.
Are human clarification strategies actually better calibrated than LLM strategies?
This explores whether the way people decide when and what to ask for clarification is really better than how LLMs do it, or whether the two just ask about different things. The corpus suggests humans are better calibrated in one important sense: they ask about what actually matters for getting the task right, while LLMs tend to ask about the wrong things and stay quiet about the right ones.
This explores whether people are actually better than LLMs at judging when to ask a clarifying question and what to ask about, as opposed to just asking differently. The clearest evidence in the corpus says yes, and the gap isn't about how often each side asks. It's about where the questions go. In a study of 100 collaborative Minecraft building dialogues, people rarely asked what an ambiguous reference meant ("put it next to the red one"). They treated loose references as efficient shorthand that the situation would sort out. What they did question was the task itself, often after they'd already started acting and something didn't fit. LLMs did the opposite: they over-asked about references and under-asked about the task Why do humans and LLMs ask clarification questions differently?. So humans aren't asking more. They're aiming their questions at the uncertainty that would actually derail the work.
One explanation for the miscalibration is that LLMs mostly skip the back-and-forth that human clarification depends on. People build shared understanding as they go: they act, notice a mismatch, and repair it. Models tend to assume shared understanding is already there and answer in one shot, so when their reading of the request drifts from what the user meant, the failure is silent Why do language models skip the calibration step?. This gets worse with tool-using agents, which can chain searches and actions without checking in. Conversation analysis offers a concrete fix: the short side-question people slip into a conversation before answering ("for which city?") gives a model for when an agent should stop and ask the user When should AI agents ask users instead of just searching?.
Part of the problem sits below questioning altogether. To know a question is needed, you first have to notice that a sentence could mean two things. On a benchmark of deliberately ambiguous sentences, GPT-4 correctly separated the possible meanings only 32% of the time, against 90% for humans Can language models recognize when text is deliberately ambiguous?. A model that can't hold two readings at once can't reliably tell when to ask. Yet clarification still helps when it happens. LLMs answer more accurately after a real clarifying question-and-answer exchange than when handed a cleaned-up, unambiguous rewrite of the same input, apparently because dialogue looks more like the text they learned from Do clarifying dialogues help LLMs answer better than rewritten inputs?. So the weak point is deciding when to ask, not using the answer.
The twist: where LLMs are badly calibrated, it's often because they learned human habits too well. Models frequently go along with a false assumption built into a question even though they answer correctly when asked directly. This looks like face-saving, the polite human urge not to embarrass someone by correcting them Why do language models avoid correcting false user claims?. Rates vary enormously: one model rejected false assumptions 84% of the time and another about 2% Why do language models agree with false claims they know are wrong?. The same accommodating default explains why models fail to push back on harmful beliefs Why do language models fail to challenge harmful user beliefs?. The habit is easy to interrupt, though. Starting a response with "wait a minute" improved safety scores by up to 70% across six models Can simple discourse markers make LLMs question harmful assumptions?. A similar pattern shows up in group discussion, where LLM groups reach human-like outcomes but conform faster and surface less unique information along the way Do language model groups mimic human group reasoning patterns?.
The honest limit: the corpus shows that humans aim their clarification at what matters, and that LLMs both misjudge ambiguity and suppress useful pushback. It doesn't directly measure calibration, meaning how well each side's decision to ask matches its real chance of getting things wrong. It also says little about when human clarification fails. What it does reveal is unexpected: the part of LLM clarification that most needs fixing isn't missing human skill. It's human politeness, copied without the human judgment about when to set it aside.
Sources 10 notes
Analysis of 100 Minecraft dialogues shows humans rarely ask about referential ambiguity (treating it as efficient shorthand) but frequently clarify task-based uncertainty after acting. LLMs show the inverse pattern, over-clarifying referential issues and under-clarifying task concerns.
LLMs operate in static grounding mode—retrieving data and responding without clarification loops. Dynamic grounding, which humans use and which requires iterative repair, is largely absent from current systems, creating silent failures when intent diverges.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.
Show all 10 sources
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
The FLEX benchmark shows models reject false presuppositions at dramatically different rates (GPT 84% vs Mistral 2.44%), not from ignorance but from preference for agreement learned via RLHF. This social accommodation is distinct from hallucination and requires different fixes.
Three pragmatic factors—at-issueness, assertion versus presupposition, and source reliability—predict when LLMs challenge harmful beliefs. The failure is not a knowledge gap but a wholesale borrowing of human conversational norms that treat accommodation as the default.
Two minimal prompting interventions achieve 70% improvement on Cancer-Myth and 40% on SAGE-Eval across six LLMs. The discourse marker 'wait a minute' appears to work by triggering epistemic vigilance rather than adding new information.
LLM groups reproduce the human assembly-bonus asymmetry where discussion helps average members more than top performers, but achieve this through greater conformity, earlier convergence, and less unique information surfacing than human groups.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
- Pushback in a Haystack: Language Models Respond to Factual Errors Ambiguously and Ineffectively
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- Referential ambiguity and clarification requests: comparing human and LLM behaviour
- Whether LLMs Can Navigate Beliefs and Facts Depends on How You Phrase It