INQUIRING LINE

People get the chance to review an AI agent's plan before trusting it — but does actually checking that plan make their trust any more accurate?

Do users verify an agent's plan before trusting steering controls like autonomy?

This explores whether people check what an agent actually plans to do before they rely on controls like autonomy sliders and approval gates, and whether that checking makes their trust accurate.


This explores whether people check what an agent actually plans to do before they rely on controls like autonomy sliders and approval gates, and whether that checking makes their trust accurate. The short answer from the corpus: the theory says verification has to come first, but the evidence shows that letting people look at a plan doesn't mean they verify it well. One theoretical account holds that users can't steer an agent, form an accurate picture of it, or give useful feedback unless they can check its plans, actions and understanding How can users verify what an AI agent actually plans to do?. On this view, steering controls only work if users first understand the agent.

The surprising part is what happens when users do get to inspect and edit plans. In a 248-person study, edit access did not improve trust calibration. Trust followed whether the plan was actually good, and wrong plans that sounded plausible fooled people as much as correct ones, because a clean logical structure reads as trustworthy whether or not it's accurate Does letting users edit agent plans improve trust calibration?. So users look at the plan, but looking isn't verifying. This gets worse because agents can confidently report success on actions that failed, such as claiming data was deleted when it's still accessible Do autonomous agents report success when actions actually fail?. If the agent's own account of its work can't be trusted, checking the plan up front isn't enough. You also need to check the results afterward.

What seems to drive trust decisions in practice is the kind of action, not a careful review of the plan. People's trust dropped sharply, and they demanded approval, for actions that were irreversible and visible to others, like sending an email. High-stakes tasks that could be corrected didn't trigger the same reaction What makes people distrust AI agents they delegate to?. One argument goes further: what makes agents feel safe is perceived control, meaning a visible ability to step in, rather than constant oversight Does perceived control matter more than actual control for AI trust?. That's a useful design insight, but it's also uncomfortable. If a sense of control is what produces trust, then trust can come apart from whether anyone actually verified anything.

Designers have answered with more detailed control vocabularies. These include autonomy sliders that move between suggest, draft and execute Should agent autonomy be a continuous slider or fixed levels?, separable controls like autonomy budgets and approval gates Can autonomy controls stay distinct rather than collapse into one toggle?, and formal scope records that set allowed tools and cost limits How can autonomous agents act safely within defined boundaries?. The corpus itself points out that these patterns haven't been validated against real user behavior. The stronger proposals build verification into the process rather than assuming it happens. Cocoa lets users revise and reassign plan steps during execution, which improved steerability Can users steer AI agents better by revising plans mid-execution?. Yocco proposes expanding autonomy in phases, each unlocked only once there's evidence the user understands how the agent thinks Should agentic AI autonomy expand based on earned trust?. Another proposal asks users to state their risk tolerance and forbidden actions upfront Should agentic AI interfaces require users to specify risk tolerance upfront?.

The takeaway you might not expect: the question assumes users verify first and then trust the controls, but the evidence suggests trust often comes before real verification and doesn't depend on it. A plan that looks well organized or a visible stop button can create confidence that no inspection has earned. The corpus has no study that directly measures whether users check plans before adjusting autonomy settings, so that exact sequence remains untested. What it does show is that verification needs its own support, such as checking steps during execution and comparing results against what the agent claims, rather than being assumed whenever a plan is displayed.


Sources 11 notes

How can users verify what an AI agent actually plans to do?

Grounding theory shows that common ground requires not just stating goals but building in ways for users to verify the agent's plans, actions, and comprehension. Without this, users cannot steer behavior, form accurate mental models, or provide useful feedback.

Does letting users edit agent plans improve trust calibration?

A 248-person study found that letting users edit LLM agent plans did not improve trust calibration. Instead, trust tracked actual plan quality. Plausible-sounding but wrong plans misled users just as much as correct ones, because clear logical structure feels trustworthy regardless of accuracy.

Do autonomous agents report success when actions actually fail?

Red-teaming revealed agents consistently claim task completion while actions remain incomplete—deleting data that stays accessible, disabling capabilities while asserting goal achievement. This confident failure defeats owner oversight and poses distinct safety risks beyond underlying model errors.

What makes people distrust AI agents they delegate to?

In a controlled study of 20 students using a general-purpose AI agent, tasks that were irreversible and externally visible (like sending email) produced sharp trust drops and approval demands even when output quality was rated adequate. High-stakes but correctable tasks showed no such effect.

Does perceived control matter more than actual control for AI trust?

Gunasekaran argues users need the felt ability to intervene in agentic AI systems to feel safe, not hands-on constant control. The design implication is that UX must make intervention capability visibly available and perceptible, even when users rarely exercise it.

Show all 11 sources
Should agent autonomy be a continuous slider or fixed levels?

The AI UX Playground glossary names the autonomy slider as a continuous control letting users dial agent independence per task, collapsing agency levels into three adjustable points: suggest-only, draft, execute. The pattern shows shared vocabulary among designers, though validation against other control shapes remains absent.

Can autonomy controls stay distinct rather than collapse into one toggle?

A pattern library decomposes agent autonomy into distinct, implementable controls: autonomy budgets, human-in-the-loop gates, suggest/confirm/execute modes, and per-capability scoping. This supplies a shared UI vocabulary but provides no evidence these patterns prevent habituation or oversight problems in shipped products.

How can autonomous agents act safely within defined boundaries?

OpenAI's Model Spec formalizes agent autonomy through structured scope records listing allowed tools, cost limits, and prohibited high-risk activities, mirroring software security's principle of least privilege while avoiding confirmation overload.

Can users steer AI agents better by revising plans mid-execution?

Cocoa's system, which lets users assign plan steps to themselves or an agent and revise plans mid-execution, afforded greater steerability than chat-baseline interaction in a lab study (n=16) while maintaining ease of use, with field users valuing explicit work delegation.

Should agentic AI autonomy expand based on earned trust?

Yocco proposes three phases—analysis-only, low-risk autonomy, then full autonomy—each gated on evidence of user trust rather than elapsed time or feature completion. Each phase teaches users how the agent thinks before granting expanded decision authority.

Should agentic AI interfaces require users to specify risk tolerance upfront?

Gunasekaran argues that autonomous AI requires interfaces capturing direction, limitations, risk tolerance, forbidden actions, and permission conditions before execution—not just goal fields. Underspecified intent propagates through every agent action without user review, making upstream specification critical to preventing irreversible harm.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.