Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions

Abstract User queries are often underspecified and may admit multiple valid interpretations. Rather than silently making assumptions about the user’s intent, a helpful assistant should surface such ambiguity by asking a clarifying question. Doing so requires two abilities: recognizing that a query is ambiguous, and acting on that recognition by seeking clarification instead of answering directly. To study these abilities, we evaluate models on ambiguous, unambiguous, and disambiguated questions in three settings: standard question answering, explicit ambiguity judgment, and behavioral analysis, where a judge model classifies responses as direct answers, refusals, or clarifying questions. We find a clear gap between recognition and behavior: models often identify ambiguity when explicitly asked to judge it, yet in the QA setting they overwhelmingly default to direct answers. Retrieved context further widens this gap by improving answerability while making models even less likely to ask clarifying questions.

Introduction. Large language models (LLMs) are now used at scale to answer questions, provide advice, and support decision making for a wide range of users (Zhang et al., 2020; Xu et al., 2023; Achiam et al., 2023; Team et al., 2023). In these interactions, user queries are often not fully specified: they may leave out crucial details, rely on unstated context, or be compatible with multiple plausible interpretations (Zhang et al., 2024). For such ambiguous queries, simply producing a single “best guess” answer forces the model to make hidden assumptions about the user’s intent, which can misrepresent the underlying task and propagate errors into downstream decisions. Instead, an effective assistant should actively seek missing information. To do so, the model first needs ambiguity awareness: the ability to recognize that a query is ambiguous or incomplete and therefore unsafe to answer directly. It should then convert this awareness into clarification behavior, by making its uncertainty visible, either by asking a clarifying question or by explicitly refusing to answer unless more information is provided, rather than silently committing to one assumed interpretation.

In this paper, we investigate ambiguity awareness and clarification behavior in LLMs. To probe ambiguity awareness, we ask LLMs to make explicit judgments about the query itself with or without retrieved context: decide whether a question is ambiguous and assign it to a category from our ambiguity taxonomy. On the behavior side, we examine whether this awareness carries over when model are directly prompted with the query: given the same questions, does the model actually act on its ambiguity judgments by asking for clarification or indicating that more information is needed.

Taken together, these findings instantiate a “knowing but not showing” pattern: models can detect ambiguity to some extent, but this awareness does not reliably surface in their default answering behavior, and is further suppressed when context is provided. From a reinforcement learning perspective (Ouyang et al., 2022), current training pipelines incentivize LLMs to reveal whatever yields high reward: producing responses that human annotators rate as helpful and useful. When expressing ambiguity awareness is not aligned with high reward, models learn to hide that awareness in their default behavior.This has two important implications. First, to make models more truthful and reliable under ambiguity, we need training objectives that positively reward expressing uncertainty, asking clarification questions, and explicitly acknowledging under-specification, rather than only encouraging answer accuracy. Second, when probing what LLMs “know” or estimating their capabilities, we must be careful about how we formulate the task: the very same model can display quite different levels of ambiguity awareness depending on whether it is prompted to answer the question or to analyze the question itself. Our study makes this discrepancy visible and points toward training and evaluation schemes that better align a model’s internal awareness with its outward behavior. Although we focus on ambiguity awareness, the same reasoning applies more broadly to other model behaviors we care about, wherever truthfully expressing what the model knows is not perfectly aligned with reward.

Related work. Ambiguous Question Answering Early work has highlighted the prevalence of ambiguity in open-domain QA. (Min et al., 2020) introduce AmbigQA, showing that over half of the questions in Natural Questions are underspecified and have multiple plausible answers. Follow-up studies confirm that even state-of-the-art models struggle on such ambiguous queries. For instance, (Wildenburg et al., 2024) and (Liu et al., 2023) find that LLMs often underperform when questions admit multiple interpretations. (Stelmakh et al., 2022) present ASQA, extending AmbigQA by providing each ambiguous question with a disambiguating context and long-form answers. In the conversational domain, (Guo et al.) presented a dataset for detecting ambiguity in multi-turn QA and generating clarifying questions. Other datasets like CambigNQ (Lee et al., 2023) target single-turn ambiguous questions, while multi-hop ambiguity has been addressed by MIRAGE (Park et al., 2025) for questions requiring reasoning across multiple documents.

Clarifying Questions One prominent approach to handle ambiguity is to ask clarifying questions. In informationseeking dialogues, researchers have built datasets where systems must pose a follow-up question to clarify user intent. (Aliannejadi et al., 2021) extend the Qulac dataset (Aliannejadi et al., 2019) with ClariQ (Aliannejadi et al., 2020), crowdsourcing single-turn conversational queries along with appropriate clarifying questions and answers. Similarly, (Kumar & Black, 2020) construct ClarQ, a clarifying question dataset derived from StackExchange posts in a QA setting. Subsequent research has treated clarifying question generation as a learning problem. For example, (Shridhar et al., 2023) train models to generate a clarification question for ambiguous inputs and use those questions to improve downstream answering via knowledge distillation. Despite progress in supervised settings, off-the-shelf LLMs rarely ask for clarification by default. (Kuhn et al., 2022) showed that an LLM can be prompted to either answer or ask a follow-up while (Deng et al., 2023) find that chat-oriented LLMs like ChatGPT often fail to ask clarifying questions for ambiguous queries unless explicitly prompted. Altogether, these prior work suggests that training and prompting methods can encourage LLM to proactively resolve ambiguity, though this behavior is not yet inherent in most models’ default responses.

Abstention and Uncertainty Another line of research addresses ambiguity by allowing the model to abstain or defer answering when uncertain (Cole et al., 2023; Kamath et al., 2020; Kadavath et al., 2022). (Shi et al., 2025) prompt an LLM to generate multiple answers to a question, and then analyze them to infer the ambiguity.

Method. 3. Experimental Setup We study these questions on AmbigQA (Min et al., 2020), a dataset containing ambiguous questions, unambiguous questions, and disambiguated rewrites of the ambiguous ones. On this dataset, we conduct three complementary evaluations. First, we measure QA accuracy on ambiguous, unambiguous, and disambiguated questions, both with and without retrieved context, to quantify how model performance differs across these categories and how the presence of context affects it. Second, we probe ambiguity awareness by asking models to decide whether each question is ambiguous, again, both with and without context. Third, we analyze clarification behavior by examining, for each question, how often the model asks a clarifying question, directly answers, or explicitly refuses and indicates that more information is needed.

We conduct all experiments on AmbigQA (Min et al., 2020), a question answering dataset built from open domain version of Natural Questions (Kwiatkowski et al., 2019) that identifies all the possible answers to an open domain question, along with the disambiguated questions. Each example consists of a question, one or more answer strings, and a set of Wikipedia passages retrieved for that question, i.e., context. For questions that are ambiguous, the dataset contains also the disambiguated version of the original ambiguous questions, as well as their respective answers. Crucially for our purposes, the dataset contains:

• Ambiguous questions, where the original user query admits multiple plausible interpretations and is annotated with multiple disambiguated question–answer pairs.

• Unambiguous questions, which have a single clear interpretation based on the retrieved context.

• Disambiguated rewrites, where each ambiguous question is rewritten into several more specific questions, each corresponding to one underlying interpretation and answer.

We evaluate each model in two explicit task settings and perform an additional behavioral analysis.

Question Answering. In the QA setting, the model is prompted with the original question from the dataset, either with or without retrieved context, and is asked to answer it directly. We then measure the correctness of the model’s response against the reference answers.

Ambiguity Judgment. In the ambiguity-judgment setting, the model is asked to determine whether a question is ambiguous or unambiguous, making this setting a binary classification task. The exact prompt is shown in Prompt B.2. This setting is designed to probe the model’s explicit ambiguity awareness.

Behavioral Analysis. In addition to task accuracy, we analyze the behavioral patterns of model outputs in both settings. In the QA setting, we classify each response as a direct answer, a clarifying question, or an explicit refusal / indication that more information is needed. In the ambiguity-judgment setting, we analyze which ambiguity categories the model assigns, how these distributions vary across models, and how they compare with human annotations.

Discussion. Our results point to several promising directions for future work. First, as highlighted throughout the paper, there is a misalignment between current reward signals and model faithfulness: models often know that a query is ambiguous, but their default behavior is to act as if it were fully specified. A natural next step is to design improved reward functions and training procedures that explicitly incentivize models to reveal what they know.

Second, we observe a striking asymmetry in how models use retrieved context. In the QA setting, context reliably improves performance, but in the ambiguity-judgment setting, the same context provides little benefit and can even make models less likely to flag questions as ambiguous. This suggests that the model are using superficial signals when deciding if a query is ambiguous rather than reasoning over the given contexts. Improving models’ ability to reason over retrieved context is an important direction for enhancing their ambiguity-awareness.