Does reasoning ability help LLMs ask better clarification questions?
Explores whether LLMs with reasoning capabilities or chain-of-thought prompting ask clarification questions more often and more relevantly. Matters because it suggests reasoning connects to metacognition and theory of mind in models.
Beyond the human/LLM mismatch covered in Why do humans and LLMs ask clarification questions differently?, this paper's second research question asks whether an LLM's ability to "ask a useful clarification question, or indeed judge when to ask a question, is based on a model's ability to simulate reasoning." Testing non-reasoning llama2:13b against chain-of-thought-prompted variants (a zero-shot "Think step by step" addition and a one-shot version with a worked example) and reasoning-tuned models (llama3.2, DeepSeek-R1), the authors report that "reasoning does appear to increase question frequency and relevancy."
The paper frames the capacity to ask a clarification question as resting on two constructs from human psychology: Metacognition (Flavell, 1979) — "reflection on the gaps in ones own knowledge" needed to know when to ask — and Theory of Mind (Premack and Woodruff, 1978), needed because "clarification may be dependent on discovering other parties['] knowledge, abilities or attitudes." It cites Kosinski (2023) for the claim that LLMs "may now be able to simulate" theory of mind as a by-product of reasoning ability, and treats the observed reasoning-to-clarification link as indirect evidence consistent with that capacity rather than a direct measurement of it.
This sits against Do larger models follow stated beliefs less often?, which finds that bigger and instruction-tuned models track a stated user belief less often, not more. The two findings are not strictly contradictory — one is about asking a question, the other about using a belief once stated — but together they warn that reasoning helps clarification and models track others' beliefs are separate claims, and that gains on one axis need not carry over to the other.
The reasoning-to-clarification link is read off a small model set (llama2, llama3.2, DeepSeek-R1) and two prompting treatments on one 100-dialogue subset, and the paper does not test whether the resulting questions were acted on correctly or improved task outcomes. The authors themselves pose the theory-of-mind framing as something to "question," not a conclusion; the finding supports a correlation between reasoning and clarification behavior, not a demonstrated mechanism by which reasoning produces theory of mind.
Inquiring lines that read this note 3
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do symbolic and semantic approaches combine in LLM reasoning? Why do LLM outputs diverge from their knowledge and reasoning? Do reasoning improvements prevent models from abstaining when needed?Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Why do humans and LLMs ask clarification questions differently?
Humans and LLMs appear to clarify uncertainty in opposite ways—humans focus on task ambiguity while LLMs flag referential confusion. Understanding this mismatch could reveal whether either approach is actually well-calibrated for dialogue.
sibling note, same paper's first research question on where humans and LLMs diverge
-
Do larger models follow stated beliefs less often?
Does model size and instruction-tuning affect whether language models prioritize user-stated beliefs over their parametric knowledge? Understanding this matters for predicting when models will accept user context versus rely on training.
a reminder that reasoning-driven gains in asking questions need not extend to using stated beliefs
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Referential ambiguity and clarification requests: comparing human and LLM behaviour
- Better Zero-Shot Reasoning with Role-Play Prompting
- Large Language Models as an Indirect Reasoner: Contrapositive and Contradiction for Automated Reasoning
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought
- Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning
- Skills-in-Context Prompting: Unlocking Compositionality in Large Language Models
- Eliciting Reasoning in Language Models with Cognitive Tools
Original note title
llm clarification questions increase in frequency and relevance under reasoning-oriented models and chain-of-thought prompting