Why do humans and LLMs ask clarification questions differently?
Humans and LLMs appear to clarify uncertainty in opposite ways—humans focus on task ambiguity while LLMs flag referential confusion. Understanding this mismatch could reveal whether either approach is actually well-calibrated for dialogue.
The paper builds a merged corpus from the Minecraft Dialogue Corpus (Narayan-Chen et al., 2019), combining reference and ambiguity annotations (MDC-R, Madge et al., 2025) with clarification-question annotations (MSDC, Thompson et al., 2024) across a shared subset of 100 dialogues. Using this corpus, the authors report "only a weak link between ambiguity and humans producing clarification questions," and "low correlation between humans and LLMs." Specifically: "Humans hardly ever produce clarification questions for referential ambiguity, but often do so for task-based uncertainty. Conversely, LLMs produce more clarification questions for referential ambiguity, but less so for task uncertainty."
The authors attribute humans' near-silence on referential ambiguity to communicative economy: "In the majority of these cases of linguistic ambiguity, the ambiguity appear deliberate in the interest of conversational efficiency," so "there may be little utility to asking a question in many of those situations." Where humans do clarify, it is mostly on the task dimension, and even then largely after the fact: the paper finds the human strategy was "to perform actions then use a clarification question to verify they were correct," rather than clarifying upfront. The LLMs tested (llama2, llama3.2, DeepSeek-R1, with and without chain-of-thought prompting) show the opposite allocation, flagging referential uncertainty literally rather than judging whether it is pragmatically safe to leave unresolved.
This complicates Do language models actually build shared understanding in conversation?: that pattern holds for task-based common ground, where this paper's LLMs under-clarify relative to humans, but not for referential ambiguity, where the same LLMs over-clarify relative to humans who tolerate it as efficient shorthand. It also sits awkwardly next to Why do language models lose performance in longer conversations?'s account of premature assumption under RLHF: premature assumption describes the task-uncertainty half of this paper's finding but not the referential-ambiguity half, where the tested LLMs instead over-ask. The companion note Does reasoning ability help LLMs ask better clarification questions? covers this same paper's second finding, on reasoning and question quality.
The corpus subset is 100 Minecraft-building dialogues, text-only, with world state encoded as JSON rather than images, and the tested models are a small, mostly open-weight set. The paper explicitly does not test "the resultant utility of a clarification question," so it establishes a mismatch in when humans and LLMs clarify, not whether either side's choices are well-calibrated; a system tuned to match human clarification timing in this domain would still need separate evidence that doing so helps.
Inquiring lines that read this note 5
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Why do LLM outputs diverge from their knowledge and reasoning? Can prompt engineering achieve genuine reasoning improvements without introducing user bias? How do language models handle contested beliefs versus facts?Related concepts in this collection 4
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Do language models actually build shared understanding in conversation?
When LLMs respond fluently to prompts, do they perform the communicative work humans do to establish mutual understanding? Research suggests they skip the grounding acts that make dialogue reliable.
contrasts: that gap describes task common ground, not the referential ambiguity this paper finds LLMs over-clarify on
-
Why do language models lose performance in longer conversations?
Does multi-turn degradation stem from fundamental model limitations, or from misalignment between what users mean and what models assume? Understanding the root cause could guide better solutions.
premature-assumption framing fits only the task-uncertainty half of this paper's mismatch
-
Does reasoning ability help LLMs ask better clarification questions?
Explores whether LLMs with reasoning capabilities or chain-of-thought prompting ask clarification questions more often and more relevantly. Matters because it suggests reasoning connects to metacognition and theory of mind in models.
sibling note, same paper's second research question on reasoning and clarification quality
-
Do LLMs recognize ambiguity but ignore it anyway?
When LLMs are explicitly asked to judge ambiguity, they often detect it—yet they default to answering ambiguous questions directly. This explores why awareness doesn't translate into clarifying behavior, and how retrieved context affects that gap.
Qualifies: LLMs often default to answering ambiguous queries directly rather than asking, complicating their referential-ambiguity clarification tendency
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Referential ambiguity and clarification requests: comparing human and LLM behaviour
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- Accommodation and Epistemic Vigilance: A Pragmatic Account of Why LLMs Fail to Challenge Harmful Beliefs
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomy
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
Original note title
humans ask clarification questions for task uncertainty while llms ask them for referential ambiguity