Could an AI figure out what you want just by watching you work, without you ever explaining it?
Can passive behavior observation replace explicit user intent articulation?
This explores whether AI systems can figure out what you want by watching what you do (your screen, your hesitations, your habits) so you don't have to spell out your goals in words, and where that approach stops working.
This explores whether AI can learn what you want by watching you instead of asking you, and where that breaks down. The corpus's short answer is that observation can do far more than most people expect, but it works best alongside asking and doesn't replace it. Watching is good at guessing what you want right now. It is much worse at knowing when a guess has drifted off course.
The strongest evidence for watching comes from systems that read your screen and infer your immediate goal. Their outputs were preferred over standard LLM responses 66–86% of the time Can systems infer what users want from watching their screen?. The trick is to aim narrow: the system doesn't try to understand your life goals, only what you're probably trying to do in the next few minutes. Other work goes beyond the screen. Gaze, typing rhythm, hesitation and speed can act as a running signal of whether you're confused, focused or stuck, so the system never has to interrupt you with a question Can AI systems read cognitive state from interaction patterns alone?. Over longer periods, agents that keep separate records of 'what happened' and 'what's generally true about this person' can build up preferences from continuous observation without ever asking Can agents learn preferences by watching rather than asking?. The payoff can be large. Agents that offer relevant information before being asked cut conversation length by up to 60%, yet this behavior is almost absent from AI benchmarks Could proactive dialogue make conversations dramatically more efficient?.
The argument for still asking comes from somewhere unexpected: conversation analysis, the study of how people actually talk. Tool-using agents tend to drift away from what the user meant by quietly chaining one action after another, and each step looks reasonable on its own. Human conversation has a built-in fix called 'insert-expansions': short side questions like 'did you mean X or Y?' that check understanding before going ahead When should AI agents ask users instead of just searching?. Without explicit check-ins, a system has nothing to catch a wrong guess before acting on it. There's also a social cost to guessing too eagerly. Agents that are smart and adaptive but lack 'civility' (respect for timing, boundaries and the user's right to steer) come across as intrusive and end up overriding what the user actually wanted How can proactive agents avoid feeling intrusive to users?.
There's a darker side that's easy to miss. The behavioral signals that let a system time its help well are the same ones that enable manipulative profiling Can AI systems read cognitive state from interaction patterns alone?. Combine that with the finding that LLMs try to persuade in almost every conversation, using logical-sounding framing that seems objective Do LLMs persuade users more often than humans do?. A system that has learned your hesitations from watching you, and that persuades by default, has a lot of quiet influence over you. When you state your intent explicitly, it does more than inform the system. It gives you a way to hold the system to what you actually said.
One reframing worth taking away: 'intent' may never have needed to be a neat label in the first place. Some dialogue systems have dropped the standard approach of sorting each message into a fixed category. Instead they turn conversation directly into actions, treating understanding as a question of 'what should happen next' rather than 'what does this sentence mean' Can command generation replace intent classification in dialogue systems?. Seen that way, watching and asking are two inputs into one ongoing guess about the next useful action. The open design question is which of the two to trust at which moment. Watching handles the routine moments; asking handles the moments where a wrong guess would be costly or hard to undo.
Sources 8 notes
Systems that passively observe user activity to infer just-in-time objectives produce outputs preferred 66–86% of the time over standard LLMs. This approach sidesteps fixed training objectives by inferring narrow, momentary goals from behavior instead.
Research shows AI systems can instrument multimodal behavioral signals (gaze, hesitation, speed) to read cognitive state during interaction, preserving flow by avoiding disruptive explicit probes. However, the same substrate enables both helpful timing and manipulative profiling.
M3-Agent demonstrates that separating episodic events from semantic knowledge in an entity-centric graph, combined with parallel memorization and control processes, allows agents to infer and act on user preferences without asking. This architecture mirrors human cognitive systems that bind disparate information about individuals across sensory modalities.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Show all 8 sources
Intelligence and adaptivity alone create socially blind agents that interrupt poorly and override user direction. The Intelligence-Adaptivity-Civility taxonomy shows civility—respecting boundaries, timing, and autonomy—is essential to making proactivity welcome rather than intrusive.
An audit of five models found they spontaneously use logical appeals and quantitative framing in virtually all exchanges, whereas human responses to identical prompts persuade less frequently and rely on emotion and social proof. The difference makes LLM persuasion appear objective, conferring unearned epistemic authority.
Rasa's dialogue understanding architecture generates domain-specific commands instead of classifying intents, eliminating annotation requirements, handling context naturally, and scaling without degradation—treating understanding as pragmatics rather than semantics.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- DiscussLLM: Teaching Large Language Models When to Speak
- Proactive Conversational Agents in the Post-ChatGPT World
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- A Survey on Proactive Dialogue Systems: Problems, Methods, and Prospects
- Proactive Conversational Agents with Inner Thoughts
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- Spontaneous Persuasion: An Audit of Model Persuasiveness in Everyday Conversations
- Do LLMs Change Their Minds Like Humans? Diagnosing Human--LLM Divergence in Single-Turn Persuasion Judgments