Should agentic AI interfaces require users to specify risk tolerance upfront?
Does designing intent frameworks that capture forbidden actions and acceptable risk before execution actually prevent harm in autonomous AI systems? And can users reliably articulate these constraints in advance?
Gunasekaran argues that as AI systems move from executing commands to acting autonomously on a user's behalf, the design problem for input shifts from capturing a goal to capturing intent in full: "direction, limitations, and exceptions; what success looks like; what failure looks like; and what must never happen." She writes that "UX design must, therefore, move beyond creating input fields and into designing intent frameworks" that prompt users to articulate "desired outcomes, acceptable risk levels, operational constraints such as time or confidentiality, actions that are explicitly forbidden, and conditions under which permissions should change."
Her reasoning: "Autonomy without context invites misinterpretation, and misinterpretation invites harm." Because the agent itself determines its steps, selects tools, and gathers data, an underspecified goal propagates through every action the agent later takes without further review from the user. She frames this as "configuration design" and "cognitive design" — the designer's job is to help a user think through the shape of a request before handing off execution, not just to collect the request. The stakes scale with irreversibility: "Good intent design prevents errors. Better intent design prevents irreversible consequences."
This supplies a concrete, pre-execution design object to a problem other frameworks address mostly from the opposite end — monitoring and correcting the agent once it is already running. Does human-in-the-loop design actually enable effective oversight? argues builders over-invest in after-the-fact intervention and under-invest in giving users the knowledge needed to oversee an agent at all; Gunasekaran's intent framework is one candidate for supplying that knowledge upstream, before the agent acts, rather than leaving the user to reconstruct it from the agent's behavior later. It also sits alongside How should AI systems be designed across different interaction phases?, which likewise groups guidance by when it applies; Gunasekaran's intent framework corresponds to the initially phase but specifies its content in more operational detail — risk tolerance, forbidden actions, and conditions for changing permissions.
The essay does not describe what an intent-framework interface would actually look like, nor test whether users can or will articulate forbidden actions and risk tolerances in advance — eliciting negative constraints ("what must never happen") is a known difficulty in requirements work generally, and the piece offers no method for prompting it reliably. It is a design argument for a missing category of interface, not a validated pattern with implementation guidance or evidence that users can supply this information well.
Inquiring lines that read this note 22
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How should organizations govern autonomy and access for agentic AI systems? Can sequences of locally safe agent actions violate system-level constraints?- How should designers balance agent ubiquity against safety and control concerns?
- How do agent goals and environments create risk beyond component errors?
- Can a boundary on permitted actions catch whether an allowed action is actually safe?
- Do deployed AI system prompts actually protect users from harmful outputs?
- Can trained safety rules inside an agent stop it from rationalizing past its own guidelines?
- How should systems formally commit to what they understand about user intent?
- Do users actually prefer confirmation prompts before high-stakes AI actions?
- How do intent frameworks compare to monitoring and intervention approaches for AI oversight?
- What makes agentic AI intent specification different from traditional goal-setting interfaces?
- Can AI shift from passive tool to active participant without creating consent problems?
- How do intervention-prediction agents differ from interface-based control strategies?
- What distinguishes intent negotiation layers from per-turn instruction assembly in AI interfaces?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does human-in-the-loop design actually enable effective oversight?
When AI agents include humans in the decision loop, does that automatically provide meaningful oversight? The research explores what specific capabilities humans need to actually oversee agent behavior.
this essay's upstream intent specification is one way to supply the knowledge the oversight fallacy says builders neglect
-
How should AI systems be designed across different interaction phases?
Can a small set of general guidelines help practitioners design better AI-infused products across different moments—initial setup, active use, error handling, and long-term adaptation?
both group design guidance by interaction phase; this essay specifies the "initially" phase's content in more operational detail
-
Can we design AI autonomy separately from capability?
Explores whether an AI agent's autonomy—how much it operates without user involvement—is a design choice independent of what the agent can actually do. This matters because it opens autonomy to governance and calibration rather than treating it as fixed by capability.
both treat the scope of an agent's autonomy as something specified upfront rather than left implicit
-
Does perceived control matter more than actual control for AI trust?
Explores whether users feel safe with agentic AI because they can intervene when needed, rather than because they constantly exercise oversight. Tests whether the psychological sense of available control, not its frequency of use, drives trust.
companion principle from the same essay, addressed as a felt property during and after execution rather than specified before it
-
Do overseers focus on agent execution or action safety?
When humans oversee AI agents, do they check whether the agent followed its plan, or whether the action itself is safe? A study of computer-use agent oversight found this distinction matters even when intervention timing is correct.
Evidence for A: overseers checking plan-compliance, not safety, shows why upfront risk-tolerance and forbidden-action specification matters
-
Do six UX patterns improve agentic AI user trust?
Victor Yocco proposes six interaction patterns spanning pre-action, in-action, and post-action phases to help users maintain control and confidence with AI agents. The question is whether these patterns actually deliver the psychological safety and trust they claim to provide.
Extends A: Yocco's pre-action intent preview and autonomy dial operationalize the intent-specification tools A calls for
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Designing for Autonomy: UX Principles for Agentic AI
- The Work Behind Delegation: A Framework for Supervising AI Coding Agents
- Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
- Levels of Autonomy for AI Agents
- The UN's AI Panel Sees Misalignment. We See Corporate (Mis)Behavior.
- Measuring AI agent autonomy in practice
- Fully Autonomous AI Agents Should Not be Developed
Original note title
Gunasekaran argues agentic AI UX must design intent frameworks that specify risk tolerance and forbidden actions before execution begins