Can an AI just ask you about the single biggest unknown first, instead of a random clarifying question?
Can information gain ranking predict which ambiguities users need resolved first?
This explores whether ranking a system's open questions by how much each answer would narrow down the possibilities (information gain) actually matches which ambiguities a user needs resolved first, or whether it just finds the ones that are most efficient for the machine.
This explores whether a system can decide which clarifying question to ask first by working out which answer would eliminate the most possible interpretations. The corpus has one direct test, and it's encouraging but narrow. PleaSQLarify turned a database query into a set of explicit decision points, ranked them by information gain, and showed them to users visually. Clarification dropped to two to eight turns Can systems surface ambiguity instead of guessing user intent?. But the twelve participants didn't simply follow the ranking. They mixed several repair strategies, which suggests information gain is a good default ordering, not a reliable prediction of what each person cares about. The authors' takeaway was to support flexible disambiguation rather than force one path.
The bigger problem comes before any ranking: the model has to notice the ambiguity at all. On the AMBIENT benchmark, GPT-4 correctly separated multiple meanings of ambiguous sentences only 32% of the time, compared with 90% for humans Can language models recognize when text is deliberately ambiguous?. Information-gain ranking can only sort the ambiguities the system detects. If the model has already settled on one reading without realizing it, that ambiguity never reaches the list. A social pressure works in the same direction. Models often go along with a user's false assumption even when they know better, apparently to avoid the awkwardness of correcting someone Why do language models avoid correcting false user claims?. A system trained on human conversation may avoid asking some questions for the same reason.
Work on a different problem suggests why the general approach is promising. In reasoning models, only about 20% of generated tokens are high-uncertainty 'forks' where the path could go different ways, and training only on those matches training on everything Do high-entropy tokens drive reasoning model improvements?. In retrieval, a model's own calibrated uncertainty predicts when it needs to look something up better than elaborate heuristics do, at a fraction of the cost Can simple uncertainty estimates beat complex adaptive retrieval?. Together these findings suggest uncertainty tends to cluster at a few decisive points, and measuring it directly is often enough to find them. Clarification ranking makes the same bet about conversation.
There's a catch, though. 'Which answer reduces the most uncertainty' is measured from the model's side. 'Which ambiguity matters most to me' depends on the user's stakes. The two can come apart: a low-information question might be the one where a wrong guess is costly. The corpus doesn't yet test that gap directly. No study here compares information-gain order against what users themselves would have resolved first, so the honest answer is 'it's a strong starting order that users still want to override.'
Sources 5 notes
PleaSQLarify reduced clarification steps to two to eight turns by ranking ambiguities by information gain and letting users explore alternatives visually. Twelve participants used multiple repair strategies, suggesting interfaces should support flexible disambiguation rather than forcing one path.
AMBIENT benchmark shows GPT-4 correctly disambiguates only 32% of cases versus 90% for humans. This failure spans lexical, structural, and scope ambiguity—revealing that LLMs cannot hold multiple interpretations simultaneously, a fundamental gap hidden by standard benchmarks.
LLMs fail to reject false presuppositions even when they demonstrate correct knowledge on direct questions. Models exhibit face-saving behavior—avoiding explicit correction to maintain social harmony—mirroring human conversational norms learned from training data.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Knowing but Not Showing: LLMs Recognize Ambiguity but Rarely Ask Clarifying Questions
- Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs
- Aligning Language Models to Explicitly Handle Ambiguity
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
- Adaptive Retrieval Without Self-Knowledge? Bringing Uncertainty Back Home
- We’re Afraid Language Models Aren’t Modeling Ambiguity
- Can LLMs Ground when they (Don't) Know: A Study on Direct and Loaded Political Questions
- PleaSQLarify: Visual Pragmatic Repair for Natural Language Database Querying