SYNTHESIS NOTE
Topics›Design Frameworks›this note

Should agentic AI interfaces require users to specify risk tolerance upfront?

Does designing intent frameworks that capture forbidden actions and acceptable risk before execution actually prevent harm in autonomous AI systems? And can users reliably articulate these constraints in advance?

Synthesis note · 2026-10-21 · sourced from Design Frameworks

Gunasekaran argues that as AI systems move from executing commands to acting autonomously on a user's behalf, the design problem for input shifts from capturing a goal to capturing intent in full: "direction, limitations, and exceptions; what success looks like; what failure looks like; and what must never happen." She writes that "UX design must, therefore, move beyond creating input fields and into designing intent frameworks" that prompt users to articulate "desired outcomes, acceptable risk levels, operational constraints such as time or confidentiality, actions that are explicitly forbidden, and conditions under which permissions should change."

Her reasoning: "Autonomy without context invites misinterpretation, and misinterpretation invites harm." Because the agent itself determines its steps, selects tools, and gathers data, an underspecified goal propagates through every action the agent later takes without further review from the user. She frames this as "configuration design" and "cognitive design" — the designer's job is to help a user think through the shape of a request before handing off execution, not just to collect the request. The stakes scale with irreversibility: "Good intent design prevents errors. Better intent design prevents irreversible consequences."

This supplies a concrete, pre-execution design object to a problem other frameworks address mostly from the opposite end — monitoring and correcting the agent once it is already running. Does human-in-the-loop design actually enable effective oversight? argues builders over-invest in after-the-fact intervention and under-invest in giving users the knowledge needed to oversee an agent at all; Gunasekaran's intent framework is one candidate for supplying that knowledge upstream, before the agent acts, rather than leaving the user to reconstruct it from the agent's behavior later. It also sits alongside How should AI systems be designed across different interaction phases?, which likewise groups guidance by when it applies; Gunasekaran's intent framework corresponds to the initially phase but specifies its content in more operational detail — risk tolerance, forbidden actions, and conditions for changing permissions.

The essay does not describe what an intent-framework interface would actually look like, nor test whether users can or will articulate forbidden actions and risk tolerances in advance — eliciting negative constraints ("what must never happen") is a known difficulty in requirements work generally, and the piece offers no method for prompting it reliably. It is a design argument for a missing category of interface, not a validated pattern with implementation guidance or evidence that users can supply this information well.

Inquiring lines that read this note 22

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How should organizations govern autonomy and access for agentic AI systems? Can sequences of locally safe agent actions violate system-level constraints? How can AI systems be proactive while respecting user intent? How can we ensure AI systems reliably follow constraints rather than circumvent them? Can humans maintain meaningful oversight as AI systems become more autonomous? How does interface design shape effectiveness in human-AI collaboration? How should control and autonomy be allocated in human-AI workplace collaboration? Does extending agent autonomy erode human capacity for meaningful oversight? How do users build accurate mental models when agent reasoning is partially hidden? How does increased AI involvement reshape human agency and accountability?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
13 direct connections · 121 in 2-hop network ·dense cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

Gunasekaran argues agentic AI UX must design intent frameworks that specify risk tolerance and forbidden actions before execution begins