Clarification studies usually test AI against a perfect, mind-reading user — does that still hold up with real, messy answers?
How does oracle-generated clarification compare to real user interactions?
This explores whether clarification studies that use an 'oracle', meaning a simulated user who always knows the intended meaning and answers cleanly, tell us how clarification works with real people, who answer in messier and more varied ways.
This explores whether clarification tested with idealized, oracle-style answers tells us what happens when real people do the answering. The short version: the corpus has no paper that runs the two side by side. It does have pieces on both sides, and together they suggest oracle setups capture what a model *can* gain from clarification. They miss how people actually behave while it happens.
On the idealized side, the strongest evidence is that a clarifying exchange helps a model more than a cleaned-up rewrite of the ambiguous request Do clarifying dialogues help LLMs answer better than rewritten inputs?. The reason is unexpected. The question-and-answer format looks more like the dialogue the model saw during pretraining, so the gain comes partly from the *form* of the exchange, not only from the information in it. Work on deciding *when* to ask stays on the model's side too. INTENT-SIM has the model imagine the different things a user might have meant, then measures how spread out those guesses are Can we predict when users need clarification questions?. That predicts when a question is worth asking better than looking at variation in the model's outputs does. Both approaches work with inferred or supplied answers, not live users.
With real people, a different picture appears. In PleaSQLarify, a tool that helps people write database queries, 12 participants resolved ambiguity in two to eight turns Can systems surface ambiguity instead of guessing user intent?. They did not follow one clean path. They switched between several ways of fixing misunderstandings, so the authors argue interfaces should support flexible repair rather than one fixed question-and-answer loop. An oracle answers exactly what is asked. Real users reframe the question, explore alternatives, and change their minds. Conversation analysis supports this When should AI agents ask users instead of just searching?. In human talk, clarifying questions are side exchanges that do more than one job: confirming intent, narrowing the scope of an answer, and making the response more useful. A model that only learns to fill in a missing slot is learning a thinner version of what people do.
There is also a hint that the format of clarification matters to real users. Studies of generated interfaces find people often prefer interactive widgets to plain text Do generated interfaces outperform text-based chat for most tasks?. Those widgets improve clarity but can make mid-course changes harder Do generated analysis UIs really work better than chat?. That trade-off only shows up when a real person is in the loop.
The takeaway: oracle studies measure the best case, where the right question meets a perfect answer. Real-user studies show clarification as a negotiation whose form, timing and medium all matter. If you want to see how big that gap is in numbers, this collection doesn't have that head-to-head comparison yet.
Sources 6 notes
LLMs answer more accurately when given clarifying questions and answers than when given minimally-edited disambiguated versions of the same ambiguous input. The gap exists because clarifying exchanges match the model's pretraining distribution better than artificially constrained rewrites.
INTENT-SIM, which clusters LLM outputs by inferred user intent and measures entropy across clusters, consistently outperforms output-entropy methods at flagging inputs worth clarifying across QA, machine translation, and natural language inference tasks.
PleaSQLarify reduced clarification steps to two to eight turns by ranking ambiguities by information gain and letting users explore alternatives visually. Twelve participants used multiple repair strategies, suggesting interfaces should support flexible disambiguation rather than forcing one path.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Research shows users strongly prefer LLM-generated interactive interfaces—dashboards, tools, animations—over text blocks, especially for structured and information-dense tasks. Structured representation and iterative refinement reduce cognitive load.
Show all 6 sources
TaskArtisan found that GUI widgets improve clarity and presentation in LLM-assisted analysis but introduce rigidity and prompting overhead. This trade-off between malleability and specification appears unavoidable: easier-to-use UIs are harder to customize mid-workflow, while flexible UIs demand engineering-style thinking from non-programmers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?
- TaskArtisan: Designing Composable Generative Widgets for LLM-Assisted Analysis
- Generative UI: LLMs are Effective UI Generators
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses
- Referential ambiguity and clarification requests: comparing human and LLM behaviour