Do LLMs recognize ambiguity but ignore it anyway?

A paper evaluating LLMs on AmbigQA, a question-answering dataset built from Natural Questions that separates ambiguous questions, unambiguous questions, and disambiguated rewrites, finds a gap between what models know about a query and what they do about it. Testing three settings — direct question answering, explicit ambiguity judgment, and behavioral classification of the QA responses as direct answers, refusals, or clarifying questions — the paper reports that models "often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers." It names this a "knowing but not showing" pattern: ambiguity awareness is present but "does not reliably surface in their default answering behavior." A second, compounding finding is that giving models retrieved context improves answerability while making models even less likely to ask clarifying questions — context makes models more confident, not more inquisitive.

The paper locates the cause in training incentives rather than a representational limit. From a reinforcement-learning-from-human-feedback perspective, "current training pipelines incentivize LLMs to reveal whatever yields high reward: producing responses that human annotators rate as helpful and useful." When expressing ambiguity awareness isn't what gets rewarded, "models learn to hide that awareness in their default behavior." The context asymmetry supports this reading: retrieved context "reliably improves performance" on direct QA, the rewarded behavior, but "provides little benefit" to ambiguity judgment and can even suppress it — evidence that models "are using superficial signals when deciding if a query is ambiguous rather than reasoning over the given contexts."

This complicates Can language models recognize when text is deliberately ambiguous?, which found GPT-4 far behind humans at generating correct disambiguations — a recognition failure. Here, ambiguity awareness is probed as an explicit judgment task rather than as disambiguation generation, and is reported as present to a meaningful degree; the shortfall is not in recognizing ambiguity but in acting on that recognition during ordinary answering. The two findings sit at different points of the same pipeline: AMBIENT shows recognition itself can fail when the task is to produce the right disambiguation; this paper shows that even where recognition succeeds, behavior doesn't follow it. The split echoes Can models identify what information they actually need?, which separates problem-solving from identifying missing information as distinct abilities — this paper draws a further, later cut, between identifying missing information (ambiguity judgment) and acting on that identification (clarification behavior), which it finds are also separable and misaligned by default. It also bears on Which clarifying questions actually improve user satisfaction?: that note concerns how well a question is designed once a model decides to ask; this paper's finding sits upstream of design, since by default most models don't ask at all, regardless of how good the question would be.

The excerpt reports findings from a single dataset built on Natural Questions, with behavioral classification performed by a separate judge model rather than human raters, and does not name which specific LLMs were tested or give pass rates. It does not test clarification behavior in open-ended or multi-turn chat, only single-turn QA with and without retrieved context. The paper's own implication is direct: closing the gap requires "training objectives that positively reward expressing uncertainty, asking clarification questions, and explicitly acknowledging under-specification, rather than only encouraging answer accuracy" — and a caution for anyone evaluating LLM capabilities more broadly, since "the very same model can display quite different levels of ambiguity awareness depending on whether it is prompted to answer the question or to analyze the question itself."