INQUIRING LINE

Why do people wait to ask questions about the goal, while AI rushes to ask which object you mean?

Why do humans and LLMs ask clarification questions differently?

This explores why people and LLMs ask for clarification about different things and at different moments, and what that difference shows about how each one handles shared understanding in conversation.


This explores why people and language models ask for clarification about different things and at different times. The corpus suggests the two have opposite instincts. In a study of 100 collaborative Minecraft building dialogues, humans rarely asked which block or which spot the speaker meant. They treated vague references as efficient shorthand and worked them out from context. Their questions came later, about the task itself, often after they had already started acting and run into uncertainty. LLMs did the reverse: they over-asked about which thing was meant and under-asked about whether they understood the goal Why do humans and LLMs ask clarification questions differently?. People clarify to keep the work on track. Models clarify the words.

The gap doesn't seem to come from models failing to notice ambiguity. Asked directly whether a question is ambiguous, models usually say yes, and then answer it anyway. Giving them retrieved documents makes this worse, because the extra context makes them more confident rather than more cautious. The likely cause is training that rewards direct answers, not an inability to see the problem Do LLMs recognize ambiguity but ignore it anyway?. The same pattern appears when a question contains a false assumption. Models often go along with the premise even when they demonstrably know it is wrong Why do language models accept false assumptions they know are wrong?. Answering is the default, and checking in has to work against it.

Looking at conversation more broadly, clarification questions are one part of what linguists call grounding. These are the small moves people use to confirm they're on the same page, like acknowledging, repairing a misunderstanding or checking in. LLMs make these moves 77.5% less often than humans. They assume shared understanding instead of building it, and fluent, authoritative wording hides the gap Do language models actually build shared understanding in conversation?. There's also a measurement problem: most human clarifications aren't phrased as questions at all. They're statements like "the red one, I think" or "wait, you mean the left side." They also happen at different levels, from "did you hear me?" to "what do you want me to do?" Why do clarification requests look different at each communication level?. A system or benchmark that only counts sentences ending in a question mark misses most of what people actually do.

The partial fixes are informative. Reasoning models and chain-of-thought prompting make clarification questions both more frequent and more relevant. One reading is that thinking step by step gives the model room to imagine what its partner knows and doesn't know Does reasoning ability help LLMs ask better clarification questions?. And when clarification does happen, it pays off more than you might expect. Models answer more accurately when given a real clarifying exchange than when given a neatly rewritten, unambiguous version of the same question, apparently because dialogue looks more like what they learned from Do clarifying dialogues help LLMs answer better than rewritten inputs?.

The reason this matters beyond tidiness: when a model skips the check-in, the reader usually can't tell. Fluent explanations raise trust in wrong answers just as much as in right ones Do LLM explanations actually help people judge answer quality?, and default explanations leave people more confident than the model's own internal probabilities justify Why do readers trust LLM explanations more than they should?. A model that asks about the wrong things therefore doesn't just waste a turn. It can quietly replace a misunderstanding you would have caught with one you won't.


Sources 9 notes

Why do humans and LLMs ask clarification questions differently?

Analysis of 100 Minecraft dialogues shows humans rarely ask about referential ambiguity (treating it as efficient shorthand) but frequently clarify task-based uncertainty after acting. LLMs show the inverse pattern, over-clarifying referential issues and under-clarifying task concerns.

Do LLMs recognize ambiguity but ignore it anyway?

Models identify ambiguity when explicitly judged but overwhelmingly answer ambiguous questions directly. Retrieved context widens this gap by increasing confidence in answers rather than prompting clarification. Training incentives reward direct answers over ambiguity acknowledgment, not representational limits.

Why do language models accept false assumptions they know are wrong?

The FLEX Benchmark shows that models reject false presuppositions at rates far below acceptable levels (GPT-4: 84%, Mistral: 2.44%), even when direct knowledge questions prove they know the correct facts. False presuppositions drive more accommodation than correct knowledge drives rejection.

Do language models actually build shared understanding in conversation?

LLMs produce grounding acts—clarifications, acknowledgments, repairs—77.5% less frequently than humans. They generate fluent responses without verifying shared understanding, relying instead on authoritative framing that masks the absence of genuine communicative calibration.

Why do clarification requests look different at each communication level?

Research maps clarification mechanisms to four levels of communication—attention, signal, meaning, action—each grounded in a different modality (socioperception, hearing, vision, kinesthetics). Most clarifications use declarative form, not questions, making them invisible to systems that detect by syntax alone.

Show all 9 sources
Does reasoning ability help LLMs ask better clarification questions?

Testing reasoning-tuned models and chain-of-thought prompting against baseline LLMs on a Minecraft dialogue task, researchers found reasoning increased both the frequency and relevance of clarification questions. The link suggests reasoning may support metacognitive reflection and simulation of others' knowledge.

Do clarifying dialogues help LLMs answer better than rewritten inputs?

LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.

Do LLM explanations actually help people judge answer quality?

A 308-person experiment found explanations increased reliance on LLM responses regardless of accuracy, while sources and inconsistent explanations reduced inappropriate trust in wrong answers. The effect stems from explanations feeling fluent and authoritative rather than being carefully evaluated.

Why do readers trust LLM explanations more than they should?

Lab experiments on MMLU and TriviaQA show readers' confidence judgments track explanation language poorly compared to the model's internal probability estimates. The gap narrows when explanations are deliberately engineered to reflect internal confidence, but natural language explanations leave readers systematically overconfident.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.