Can we design AI autonomy separately from capability?
Explores whether an AI agent's autonomy—how much it operates without user involvement—is a design choice independent of what the agent can actually do. This matters because it opens autonomy to governance and calibration rather than treating it as fixed by capability.
The paper argues that an AI agent's level of autonomy — "the extent to which it is designed to operate without user involvement" — is a deliberate design decision that developers can set independently of the agent's capability or operational environment, and proposes five levels defined by the role a user takes: operator, collaborator, consultant, approver, and observer. It states plainly that "a capable agent...can still act with a low level of autonomy if it is required to consult its user before taking each action," treating autonomy and capability as orthogonal properties rather than the same dial.
The reasoning rests on separating two terms the paper says are often conflated: agency ("the capacity to formulate an intention...and carry out that action") and autonomy ("the extent to which an AI agent is designed to operate without user involvement"). An agent can have low agency (few tools, like a calculator-only API) and high autonomy (it runs unsupervised in the background), or the reverse — so autonomy is not simply what capability allows but a separate lever developers pull through "a set of model- and user interface-based constraints." The five user roles mark points on that lever, each defined by the form and degree of control a user retains, from directing each action (operator) to only monitoring outcomes after the fact (observer). The paper's proposed application, AI autonomy certificates issued by a third party to cap "the maximum autonomy level at which an agent is permitted to operate," treats the levels as governable commitments rather than descriptive labels, with uses it names in risk assessment, safety-framework design, and multi-agent communication.
The paper's own citation trail ties it directly to Does AI risk increase with the autonomy we give it? — it cites that argument ("researchers have even argued that fully autonomous agents should not be developed") as the backdrop its levels are meant to govern, substituting a calibrated spectrum for an all-or-nothing ban. Structurally it parallels Does machine agency exist on a spectrum rather than binary?, another five-level taxonomy of agency — but where that spectrum levels by degree of autonomy and coordination (passive to cooperative), this framework levels by the user's role in the interaction, a difference in what the levels are indexed to rather than how many there are. Its insistence that autonomy and capability should be "measured...separately" echoes Does a single benchmark score actually predict agent readiness?, extending the separate-axes argument outward: not just that capability itself decomposes into axes, but that autonomy is a further axis outside capability altogether. The approver and observer roles at the low-control end of this framework also sit against two empirical notes: Where do user values break down in agent supervision? finds that supervision is exactly where value satisfaction breaks down, and Does granting agents more autonomy undermine human oversight? suggests the approver role this paper designs for may itself erode with use.
The excerpt does not validate the five levels empirically — no user study, deployment data, or inter-rater test of where real agents or products (Operator, Devin, Deep Research) actually fall on the scale accompanies the taxonomy, and the autonomy-certificate mechanism is described only as a "potential application" with "one practical implementation" gestured at but not specified. The evaluation methods for measuring autonomy itself are likewise left as "early ideas," not a working protocol. The paper offers a conceptual scaffold for the field to adopt and test, not yet evidence that user role is the right or only axis along which to design agent autonomy.
Inquiring lines that read this note 17
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
Does expanding agent autonomy undermine human oversight capacity?- How should AI oversight work if agency support doesn't follow capability gains?
- Does giving AI systems more autonomy erode human capacity to oversee them?
- How does autonomy erosion affect overseer capability versus overseer judgment criteria?
- What does autonomy mean in the design of AI agents?
- Can third-party autonomy certificates effectively cap how independently agents operate?
- Does agency spectrum differ from autonomy spectrum in meaningful ways?
- Do users prefer per-action autonomy settings over single global autonomy controls?
- What defines and measures AI agent autonomy across different products?
- How do power users differ in granting autonomy to AI agents?
- How do action-space and autonomy interact as separate governance levers?
- What five levels of autonomy can users set for AI agents in practice?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does AI risk increase with the autonomy we give it?
Explores whether the risks posed by AI agents scale monotonically with the level of autonomy they're granted, and what the tradeoffs are between human control and agent independence.
the paper cites this argument directly as the position its governed spectrum is designed to answer instead of an outright ban
-
Does machine agency exist on a spectrum rather than binary?
Rather than viewing AI as either autonomous or controlled, does machine agency actually operate across five distinct levels from passive to cooperative? Understanding this spectrum matters because it shapes how users calibrate trust and control expectations.
a parallel five-level taxonomy, but indexed to degree of autonomy/coordination rather than user role
-
Does a single benchmark score actually predict agent readiness?
Single-axis benchmarks rank models by one capability—like task success—but ignore privacy, duration, operating mode, and ecosystem fit. Can one number really capture what matters for deployment?
shares the separate-axes argument, extended here to autonomy as an axis independent of capability
-
Where do user values break down in agent supervision?
When people use AI agents, their values tend to align with delivered outputs but conflict during oversight. What explains this gap, and what does it reveal about delegation design?
empirical counterpart: supervision, the low end of this framework's control scale, is where value satisfaction breaks down
-
Does granting agents more autonomy undermine human oversight?
Explores whether the design of autonomous AI systems—by giving agents greater independence—actually weakens the human overseer's ability to catch problems. Matters because oversight is a key safeguard against AI failures.
complicates the approver/observer roles by showing the oversight capacity they depend on can erode with use
-
How should AI agents balance autonomy with user oversight?
How should decision authority between AI agents and users be allocated so it matches what users expect and can actually oversee? This matters because output alignment alone doesn't ensure users retain meaningful control.
Extends: proposes autonomy alignment — matching decision authority to user expectations across model, harness, and interaction levels — as A's five-level target
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Levels of Autonomy for AI Agents
- Human Attribution of Causality to AI Across Agency, Misuse, and Misalignment
- Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents
- Visibility into AI Agents
- Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
- Fully Autonomous AI Agents Should Not be Developed
- Agents Are Not Enough
- Explaining AI Agents Through Execution Traces
Original note title
agent autonomy splits into five user-centered levels — operator, collaborator, consultant, approver, observer