Should an AI assistant double-check that it actually understood what you asked for, not just wait for you to catch its mistakes?
Do users need to verify agent plans, or can agents also verify user intent?
This asks whether checking that the agent and the user understand each other only runs one way (the person inspects the agent's plan), or whether the agent should also check that it has understood what the person actually wants.
This asks whether checking only runs one way, with the person inspecting the agent's plan, or whether the agent should also check that it understood what the person wants. The corpus says it has to run both ways. It also suggests the one-way version, where the user reviews the plan, works less well than you'd expect. In grounding theory, shared understanding isn't created by stating a goal. Both sides need ways to confirm they read each other correctly. Users need ways to check the agent's plans, actions and understanding, or they can't steer it or give it useful feedback How can users verify what an AI agent actually plans to do?.
Letting people review plans has a problem, though. In a 248-person study, letting users edit agent plans didn't make their trust any better matched to how good the plans were. Wrong plans that sounded plausible won as much trust as correct ones, because a clearly structured plan looks right whether or not it is Does letting users edit agent plans improve trust calibration?. The same study found that editing a plan before it ran helped weak plans but made strong plans worse. Getting involved while the plan was running improved accuracy across all the tasks Does user editing of AI plans always improve performance?. Cocoa points the same way. It lets people revise a plan step by step as it runs, or take a step for themselves, and users could steer it better than a plain chat interface without any extra effort Can users steer AI agents better by revising plans mid-execution?. So the moment of checking matters more than whether checking happens at all.
What about the agent checking the user? Conversation analysis offers a model. When people talk, they sometimes pause the main exchange for a short side question, such as "for which dates?" or "do you mean X or Y?". Researchers call this an insert-expansion. It gives agents a principled way to decide when to stop and ask the person instead of chaining tool calls on a guess. Agents that drift away from what the user meant usually do it by quietly guessing, not by refusing to ask When should AI agents ask users instead of just searching?. A related finding is that remembering a user's saved preferences is a separate skill from finishing the task. An agent can complete the task and still ignore what you told it last time Do phone agents succeed at all three critical tasks equally?.
The two directions can share the work. OpenAI's ScopeOfAutonomy record has the user state their intent up front: which tools the agent may use, how much it may spend, what it must never do, and when the job counts as finished. The agent then acts inside those limits without asking for approval at every step How can autonomous agents act safely within defined boundaries?. After the action, plain-language "action receipts", modeled on bank notifications, let ordinary users check what actually happened Should agent accountability be designed for end users instead of engineers?. Taken together, these suggest a sequence. The user sets boundaries, the agent asks when it isn't sure, the user steps in during the run, and a receipt confirms the result.
One caution: checking intent one step at a time can miss intent that is spread across steps. SafeFlow shows that a harmful goal can be split into subtasks that each look harmless, so an agent checking each request separately would approve every one Can task decomposition hide harmful intent across agents?. Intent sometimes exists only in the whole plan, not in any single step. The corpus doesn't yet have direct studies of agents that model user intent across a whole task.
Sources 9 notes
Grounding theory shows that common ground requires not just stating goals but building in ways for users to verify the agent's plans, actions, and comprehension. Without this, users cannot steer behavior, form accurate mental models, or provide useful feedback.
A 248-person study found that letting users edit LLM agent plans did not improve trust calibration. Instead, trust tracked actual plan quality. Plausible-sounding but wrong plans misled users just as much as correct ones, because clear logical structure feels trustworthy regardless of accuracy.
A study of 248 users found that editing AI plans before execution helped low-quality plans but degraded already-good ones. However, user involvement during execution reliably improved accuracy across all tasks, suggesting the intervention stage and interface design shape whether human oversight helps or hurts.
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Show all 9 sources
MyPhoneBench demonstrates that task success, privacy-compliant completion, and saved-preference reuse are statistically distinct capabilities with no model dominating all three. Success-only rankings do not predict privacy or preference performance.
OpenAI's Model Spec formalizes agent autonomy through structured scope records listing allowed tools, cost limits, and prohibited high-risk activities, mirroring software security's principle of least privilege while avoiding confirmation overload.
The AI UX Playground glossary defines action receipts as bank-notification-style records for task delegators, not engineers. This positions transparency as a UX surface for non-technical users rather than an audit trail, with receipts operating after actions complete to enable verification.
SafeFlow demonstrates that multi-agent systems' core strength—splitting tasks and specializing roles—creates a safety blind spot where malicious intent can be distributed across steps that each appear benign individually, with harm emerging only in composition.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant
- Cocoa: Co-Planning and Co-Execution with AI Agents
- When Are LLM Inferences Acceptable? User Reactions and Control Preferences for Inferred Personal Information
- Overseeing Agents Without Constant Oversight: Challenges and Opportunities
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments