Does perceived control matter more than actual control for AI trust?
Explores whether users feel safe with agentic AI because they can intervene when needed, rather than because they constantly exercise oversight. Tests whether the psychological sense of available control, not its frequency of use, drives trust.
Gunasekaran argues that users of agentic AI systems do not need constant, hands-on control to feel safe — they need the sense that they could intervene if something went wrong. She writes: "Users do not require constant control to feel safe. They require perceived control. The difference between these is subtle but profound." A system that permits intervention, even when the user rarely exercises it, "feels trustworthy," while one the user "cannot interrupt, override, or question quickly becomes threatening, regardless of how accurate or efficient it may be."
Her reasoning is psychological rather than technical: "humans measure safety by their ability to intervene when something feels wrong." The design implication she draws is that UX for agentic AI must "design the experience of control as carefully as we design an AI's autonomy" — control becomes a felt, perceptible property of the interface, not merely a backend permission or kill switch that exists but stays invisible to the user. This reframes human control as a UX design object in its own right, distinct from how much control a user actually ends up exercising.
This sharpens the control dimension that recurs across adjacent frameworks. The What UX principles do workplace users want in AI agents? study finds users weight human control heavily but does not explain why a rarely-used override still matters; Gunasekaran's psychological framing supplies that mechanism — it is the felt availability of intervention, not its frequency, that produces trust. It also complicates Can automation and human control increase together?: where Shneiderman treats control as a structural axis to be maximized alongside automation, Gunasekaran treats control as a perception to be designed, which could in principle be engineered even where the actual intervention capability behind it is limited.
The essay does not test this claim empirically — it offers no study, user research, or data on what specifically makes control "feel" available, nor how designers might measure perceived versus actual control, or where the two diverge in ways that mislead users (a sense of control without the real capacity to intervene could be comforting but false). The piece is a practitioner argument, not a validated finding; it points to a hypothesis designers might test rather than a settled design principle.
Inquiring lines that read this note 19
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do confidence signals and expertise language mislead users' trust calibration?- How do stochastic outputs create mental model and trust problems for users?
- Can ambient visual signals replace explicit negotiation surfaces for AI trust?
- How do users conflate intuitive trust with reasoned evaluation of AI outputs?
- What counts as clear data that users actually trust an agentic system?
- What makes users withdraw trust from AI agents they initially delegated to?
- What role does user trust play when evaluating unpredictable AI systems?
- How can designers make control feel available without constant user oversight?
- Where does perceived control diverge from actual intervention capability in AI interfaces?
- Why do users weight human control so heavily in workplace AI agent preferences?
- How do workplace users prioritize safety and control in AI agent design?
- Does ceding decision control to AI impose a psychological cost on users?
- Can autonomy and perceived control both increase together in agentic system design?
- Can autonomy granted to users ever be revoked if trust breaks down?
Related concepts in this collection 5
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
What UX principles do workplace users want in AI agents?
A multi-method study asked business users what design principles matter most for human-AI collaboration at work. The findings prioritize oversight and reliability over convenience.
shares the control principle but doesn't explain why control that's rarely exercised still builds trust
-
Can automation and human control increase together?
Does the long-standing trade-off between automation and human control reflect an unavoidable constraint, or a design choice? This matters because it determines whether systems can be both powerful and trustworthy.
treats control as a structural axis to maximize, where this essay treats it as a felt perception to design
-
Can we design AI autonomy separately from capability?
Explores whether an AI agent's autonomy—how much it operates without user involvement—is a design choice independent of what the agent can actually do. This matters because it opens autonomy to governance and calibration rather than treating it as fixed by capability.
both treat control as a design choice, but this essay argues the felt experience of control, not the formal level, drives trust
-
Should agentic AI interfaces require users to specify risk tolerance upfront?
Does designing intent frameworks that capture forbidden actions and acceptable risk before execution actually prevent harm in autonomous AI systems? And can users reliably articulate these constraints in advance?
companion principle from the same essay, addressed before execution rather than during it
-
How should AI agents balance helpfulness with staying out of the way?
Microsoft Design explores whether AI agents should proactively intervene or remain invisible until needed, and how to make them discoverable without being intrusive or replacing human judgment.
Extends A: Microsoft's uncertainty-paired transparency-and-control principle offers a concrete design instantiating perceived control over constant oversight
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Designing For Agentic AI: Practical UX Patterns For Control, Consent, And Accountability
- Designing for Autonomy: UX Principles for Agentic AI
- The Oversight Fallacy: Why AI Agents Require More than Humans-in-the-Loop
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent
- The UN's AI Panel Sees Misalignment. We See Corporate (Mis)Behavior.
- Human oversight of agentic systems in practice: Examining the oversight work, challenges, and heuristics of developers using software agents
Original note title
Gunasekaran argues perceived control, not constant control, determines whether agentic AI feels trustworthy