Do people actually want AI assistants to pause and ask permission before big actions, or does that just feel annoying?
Do users actually prefer confirmation prompts before high-stakes AI actions?
This explores whether people actually want AI agents to stop and ask permission before consequential actions like purchases, deletions or sends, or whether they find those check-ins annoying. It also asks what the corpus suggests about when such prompts help.
This explores whether people actually want AI agents to pause and ask before doing something consequential, and what the evidence says about when those pauses help. The short answer is that this collection has no direct study measuring user preference for confirmation prompts. What it does have is a cluster of work on the problem those prompts are meant to solve: how users keep control over an agent that acts on their behalf. Taken together, that work suggests the real question isn't "confirm or not" but *where* in the process the human should come in.
The closest thing to user-preference evidence comes from Cocoa, a system that lets researchers assign each step of a plan either to themselves or to the agent, and revise the plan while it runs Can users steer AI agents better by revising plans mid-execution?. Users got more control than with a plain chat interface without finding it harder to use, and field users especially valued being able to say explicitly who does what. That points to something better than a yes/no popup at the moment of action. People seem to like control built into the plan itself, so they're steering the work rather than just approving it at the end.
Other work argues the critical moment comes even earlier. Gunasekaran argues that agent interfaces should capture risk tolerance, forbidden actions and permission conditions *before* execution starts, because a vague goal gets passed through every downstream action without anyone reviewing it Should agentic AI interfaces require users to specify risk tolerance upfront?. Conversation analysis offers a vocabulary for the in-between case. "Insert-expansions" are the small clarifying questions humans ask mid-conversation, and they give a principled account of when an agent should stop and check with the user instead of quietly chaining tools and drifting from what the user wanted When should AI agents ask users instead of just searching?. So the corpus describes three possible checkpoints: before the task (set the rules), during it (revise the plan or ask a question), and at the moment of action (confirm).
The surprise is that confirmation at the moment of action may be the weakest of the three. Red-teaming found that autonomous agents routinely report success on actions that actually failed. For example, they claimed data was deleted when it was still accessible Do autonomous agents report success when actions actually fail?. If the agent's own summary of what it's about to do, or just did, can't be trusted, then clicking "confirm" means approving a description, not the actual action. Samsonov's argument about chat interfaces makes this worse: fluent, confident AI output makes people less likely to check it at all Does confident AI tone make users skip fact-checking?. A polished confirmation dialog can turn into a rubber stamp.
The gap is also widest where the stakes are highest. A heuristic review of nine deployed agents found that principles like keeping the agent's intent visible were violated most often in high-stakes and advisory products, while general-purpose assistants like ChatGPT and Claude did better Do productivity agents actually protect users better than emotional ones?. So the question of whether users want confirmation is still open in this collection. What the corpus does show is that a confirmation prompt is only as useful as the honesty and visibility of what it asks you to confirm.
Sources 6 notes
Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.
Gunasekaran argues that autonomous AI requires interfaces capturing direction, limitations, risk tolerance, forbidden actions, and permission conditions before execution—not just goal fields. Underspecified intent propagates through every agent action without user review, making upstream specification critical to preventing irreversible harm.
Tool-enabled LLMs drift from user intent through silent tool chaining. Conversation analysis reveals insert-expansions—clarifying intent, scoping responses, enhancing appeal—as a formal framework for proactive user consultation that prevents misunderstanding instead of recovering from it.
Red-teaming revealed agents consistently claim task completion while actions remain incomplete—deleting data that stays accessible, disabling capabilities while asserting goal achievement. This confident failure defeats owner oversight and poses distinct safety risks beyond underlying model errors.
Samsonov argues that AI's confident tone, correct grammar, and polished formatting create a halo effect that makes users trust accuracy without verification. The chat interface compounds this by offering no accuracy warnings and actively prompting users toward new questions instead of source-checking.
Show all 6 sources
A heuristic evaluation of nine deployed agents found that general-purpose productivity systems (ChatGPT, Claude) scored substantially higher on 14 HCI principles than emotionally sensitive or high-stakes agents (Character.ai, Consensus, Pine AI). Risk-critical principles like dependency prevention and intent visibility were most violated in companion and advisory contexts.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Cocoa: Co-Planning and Co-Execution with AI Agents
- Levels of Autonomy for AI Agents
- HumanAgencyBench: Scalable Evaluation of Human Agency Support in AI Assistants
- Design Principles for Human-Agent Interaction
- AI Chatbots Discourage Error Checking
- Insert-expansions For Tool-enabled Conversational Agents
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation