Should an AI assistant guess what you want and just act, or pause and ask when it's not sure?
What does it mean to infer user goals probabilistically under uncertainty?
This explores what it looks like for an AI system to treat a user's goal as an educated guess with a confidence level, rather than a fixed instruction, and how that guess should shape when it acts, when it asks, and what it asks.
This explores what it means for an AI to treat a user's goal as a best guess with a confidence level attached, rather than something it already knows, and how that uncertainty should decide whether it acts or asks. The idea goes back to Horvitz's 1999 mixed-initiative principles How should agents decide when to act versus ask users?. An assistant should keep several possible goals in mind, each weighted by how likely it is. It should act on its own only when the expected benefit is greater than the cost of being wrong, and it should use conversation to settle the doubt when the odds are too close to call. That turns "should the agent act or ask?" from a yes-or-no switch into a calculation. The same calculation also tells the agent when to interrupt and when to stay quiet.
If uncertainty is the problem, asking good questions is how you reduce it, and the corpus has some precise ways of doing that. One approach simulates the answers a user might give to each candidate question. It then picks the question whose answers would shrink the agent's uncertainty the most How can models select the most informative question to ask?. Another shows that about ten well-chosen questions can be enough to work out a person's individual preferences, treated as a mix of a few shared basic tastes Can user preferences be learned from just ten questions?. Conversation analysis offers a more human version of the same idea. "Insert-expansions" are the short clarifying side-questions people ask mid-conversation before answering. They give agents a model for checking in before they quietly chain tool calls in the wrong direction When should AI agents ask users instead of just searching?.
The surprising part is that this belief-updating math works as a training signal as well as a decision rule. In a 20 Questions setup, an agent was rewarded each turn by how much its confidence in the right answer went up Can an agent's own beliefs guide credit assignment without critics?. Small models trained this way matched or beat larger ones, with no separate judge model needed. A model that is good at narrowing down what you want can therefore learn that skill by watching its own guesses improve.
Today's models mostly skip this step. In UserBench, where users reveal their goals bit by bit, models fully matched what users wanted only 20% of the time. Even the best models found fewer than 30% of preferences by asking agents-align-with-all-user-intents-only-20-percent-of-the-time-even-best-m. Their typical mistake is assuming too early instead of asking. One cause may be that models look socially skilled only when they can see everything. When one model plays every side of a conversation, it does fine, but it breaks down once each side holds private information. That gap is exactly what goal inference is meant to cover Why do LLMs fail when simulating agents with private information?.
The same question also applies on a larger scale, to recommendation systems. LLMs can read activity logs and pull out long-running goals, such as "designing hydroponic systems for small spaces," that collaborative filtering (recommending based on what similar users liked) misses Can language models discover what users actually want from activity logs?. LLMs fine-tuned on psychology experiment data also predict human decisions better than traditional cognitive models do Can language models learn to model human decision making?. Benedict Evans goes further and argues that platforms could simply rent this "why does the user want this" inference through an API instead of collecting their own behavioral data Can LLMs infer user needs better than owned behavioral data?. However, the same note reports that LLMs make up 35–49% of their claims about user attributes. That is the risk of inference that never admits its uncertainty, and it is why the probabilistic approach treats "I'm not sure, so I'll ask" as a core feature rather than a failure.
Sources 10 notes
Horvitz's twelve mixed-initiative principles argue that agents should infer user goals probabilistically, weigh expected costs and benefits before acting, time actions to minimize distraction, and use dialog to resolve uncertainty. This framework treats deferral as a design problem solvable through principled reasoning rather than binary agent-versus-user choices.
UoT combines uncertainty-aware scenario simulation with information-gain scoring and reward propagation to identify questions whose possible answers maximally reduce diagnostic uncertainty—providing a principled mechanism for specific, high-value clarification rather than generic prompts.
PReF learns base reward functions from preference data, then uses active learning to select maximally informative questions that reduce coefficient uncertainty. Users can be personalized via inference-time reward alignment without weight modification.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Show all 9 sources
Research shows LLMs perform well when one model controls all interlocutors but fail systematically when agents possess private information. This reveals that apparent social competence relies on grounding work that models skip in omniscient settings.
66% of users pursue valued interest journeys lasting over a month, described in specific phrases like 'designing hydroponic systems for small spaces.' LLM-powered journey discovery bridges the semantic gap that collaborative filtering cannot reach, operating at user-level granularity with persona-level precision.
LLMs finetuned on psychology experiment data predict human behavior more accurately than theory-driven models in decision tasks, capture individual differences in their embeddings, and transfer learning across tasks without task-specific design.
Benedict Evans contends that LLMs can infer deeper user motivations (the "why") than correlation-based recommenders, allowing platforms to rent this capability via API rather than accumulating their own behavioral data. However, research shows LLMs fabricate 35–49% of user attribute claims, undermining confidence in their inferred understanding.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Large Language Models for User Interest Journeys
- The Personalization Mirage: How LLMs Fabricate User Profiles, and Why Self-Monitoring Misleads
- Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
- User-LLM: Efficient LLM Contextualization with User Embeddings
- Intrinsic Credit Assignment for Long Horizon Interaction
- Language Model Personalization via Reward Factorization
- Determinants of LLM-assisted Decision-Making
- Uncertainty of Thoughts: Uncertainty-Aware Planning Enhances Information Seeking in Large Language Models