Levels of Autonomy for AI Agents

Autonomy is a double-edged sword for AI agents, simultaneously unlocking transformative possibilities and serious risks. How can agent developers calibrate the appropriate levels of autonomy at which their agents should operate? We argue that an agent’s level of autonomy can be treated as a deliberate design decision, separate from its capability and operational environment. In this work, we define five levels of escalating agent autonomy, characterized by the roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer. Within each level, we describe the ways by which a user can exert control over the agent and open questions for how to design the nature of user-agent interaction. We then highlight a potential application of our framework towards AI autonomy certificates to govern agent behavior in single- and multi-agent systems. We conclude by proposing early ideas for evaluating agents’ autonomy. Our work aims to contribute meaningful, practical steps towards responsibly deployed and useful AI agents in the real world.

Introduction. The earliest grand visions for AI promised intelligent agents that can learn our preferences, autonomously perform tasks on our behalf, and transform the way we consume information [24, 26]. Today, rapid growth in the capabilities of frontier generative AI models—in particular, large language models (LLMs)—renders these visions more achievable than ever. These models, which have demonstrated sophisticated understanding of natural language and our visual world, can equip AI agents with impressive autonomous capabilities, such as proactively resolving GitHub issues [18, 42], shopping online [43], and generally using computers in a human-like manner [1, 31]. Excitement over autonomous agents1 is pervasive—autonomy is often touted as a new paradigm shift in AI by companies, developers, and the press [28]. This enthusiasm has been reinforced by product releases such as OpenAI’s Operator [31], Cognition’s Devin [39], and Google’s Deep Research [11], accelerating agents’ inclusion into society.

While autonomy can unlock innovative applications that amplify the benefits of AI, it can also do the same for AI’s negative consequences, raising significant concerns about AI risks. Scholars contend that it is simultaneously more important and more difficult to anticipate harms from autonomous AI, especially as accountability for AI actions becomes harder to trace [6]. Such harms may manifest immediately (e.g., scams [12] and leaking private information [23]) or more gradually (e.g., human deskilling and loss of critical thinking [8] and societal disempowerment [22]). In Anthropic’s Responsible Scaling Policy framework [2], systems that show “low-level autonomous capabilities” are classified as ones that pose “significantly higher risk.” Indeed, researchers have even argued that fully autonomous agents should not be developed [28]. Despite these concerns, research and development of agents continue to focus on automating tasks, likely in part due to how the current evaluation paradigm of using benchmarks narrowly focuses on autonomous task completion accuracy [19].

In this essay, we argue that rather than treating autonomy as an inevitable consequence of increasing agent capability, autonomy can instead be a deliberate design decision made by agent developers. That is, given a fixed set of capabilities and a fixed operational environment, developers can make intentional choices about the level of autonomy within which an agent operates, and guide the design of their agent with those choices. But what do these levels entail, and how might they be applied in practice?

We first introduce a five-level framework (Table 1) for agent autonomy. Key to our framework is its usercentered view. We define an AI agent’s autonomy as the extent to which it is designed to act without user involvement, and thus build our framework around five roles a user can take when interacting with an agent: operator, collaborator, consultant, approver, and observer. Importantly, while an agent’s capabilities may influence its autonomy [29] (or agency [28]), our framework views autonomy as a property that can also be designed independently of capability. For instance, a capable agent—one that performs well on evaluation benchmarks—can still act with a low level of autonomy if it is required to consult its user before taking each action. Recognizing autonomy as a double-edged sword that yields both exciting benefits and consequential risks, our framework is meant for agent developers and relevant stakeholders to make informed calibrations of agent autonomy based on target use cases and desired user experiences.

We then explore one potential application of our framework by proposing an agent governance mechanism which we call autonomy certificates. These certificates are issued by a third-party governing body to agent developers to communicate behavioral characteristics of an agent to other developers and agents. We highlight the utility of autonomy certificates for risk assessment, safety framework design, and multi-agent systems engineering, and discuss one practical implementation.

Given our framework and its potential application via autonomy certificates, a key question remains: how can we evaluate an agent’s autonomy? Current agent evaluation practices primarily rely on capability benchmarks [19], but if autonomy is a design decision that can be considered independently of capability, we should also measure the two separately. We conclude with ideas for evaluation designs that allow a user to collaborate with the agent on evaluation tasks to better probe the nature of an agent’s autonomous behaviors.

Method. 2 Terminology AI agents. In their seminal AI textbook, Stuart Russell and Peter Norvig defined an “agent” as anything that can perceive its environment through sensors and execute actions in its environment through effectors [34]. Put simply, an agent is characterized by the environment it operates in and the set of possible actions it can take within that environment [16]. Examples of actions for an AI agent include fetching information from a website or clicking a button in a user interface on behalf of the human user. Software that allows AI agents to execute these actions and modify their environment is commonly referred to as “tools” [35]. Kapoor et al. [19] simply refer to them as “compound AI systems.” In our essay, we extract common threads from these definitions and refer to AI agents as compound software systems, inclusive of one or more AI models, that operate within an environment and take actions within it.

Users. We define an agent’s user to be an entity, human or AI, who issues an initial request for the agent’s services. In human-AI systems, the user is typically a human, while in multi-agent systems, users are commonly other AI agents. An agent can simultaneously be a user and be used. Consider a simple multi-agent system where a human user converses with a chatbot agent that can call upon an agent with specialized medical knowledge. When the human issues a medical query to the chatbot, they become the chatbot’s user. The chatbot can then become a user of the medical agent by initiating a request for relevant medical information that helps answer the query. Note that, depending on the autonomy of the medical agent, it may proactively follow up with clarification questions in response to the initial request, but this proactivity does not make the medical agent a user of the chatbot agent because it did not initiate the original request.

Agency. A wide range of academic disciplines, including philosophy, psychology, economics, and cognitive science, have long wrestled with complex ideas in (human) agency. Scholars generally understand agency to be “the capacity to act intentionally” [37]. Intentionality stems from the presence of a reason for a particular action. The reason may be derived from extrinsic observations of the environment and/or intrinsic values. Different disciplines may be concerned with different sources of intentionality. For example, in philosophy, an agent’s actions are often informed by moral responsibilities [9], whereas economics may assume an agent’s actions are rational, such that it maximizes its individual utility. In our essay, we keep our understanding of agency broad so it is applicable to both humans and AI. Agency is present as long as there exists the capacity to formulate an intention2 for an action and carry out that action.

Autonomy. In robotics and human-robot interaction, autonomy is broadly interpreted as “the ability to operate without a human operator for a protracted period of time” [3, 20]. We build off this definition and use autonomy to refer to the extent to which an AI agent is designed to operate without user involvement. Per our earlier definition, “user” can be a human or another AI agent. We use “designed to operate” because, as we later argue, autonomy is a design decision accompanied by a set of model- and user interface-based constraints that developers can use to shape agent behavior. “Involvement” is multifaceted and includes a spectrum of actions from direct control to light supervision. The multifaceted nature of involvement forms the foundation of our autonomy framework we will introduce later.

Why do we disambiguate autonomy and agency? We see them as two distinct levers for agent governance. An agent may have low agency if the set of tools it has access to is limited (e.g., it can only call a calculator API but cannot browse the web), thereby restricting its capacity to carry out an intentional action. The same agent may have high autonomy if it runs continuously in the background of a user’s application and processes high volumes of calculations without user supervision.

Conclusion. Autonomy is simultaneously one of the most exciting and most concerning properties an AI agent could possess. On one hand, it can unlock transformative and innovative uses of AI and distribute its benefits at unprecedented scales. On the other hand, autonomy introduces new classes of AI risks while exacerbating existing ones. With autonomy as a double-edged sword, how can AI agents be designed to be useful, reliable, and safe across society?

In this essay, we argued that autonomy can be a design decision that can be considered independently of the agent’s capabilities and operational environment. While capability and environment can clearly influence autonomy, viewing autonomy as an independent design decision can have several benefits. Given a set of capabilities and a target deployment environment for their agent, developers can craft more refined user experiences to improve human-AI collaboration, develop more effective communication protocols with other agents in multi-agent systems, and systematically reason about possible failure modes and resolution approaches. We introduce a framework for five levels of agent autonomy as a step towards this vision. We adopt a user-centric approach in our framework, centering our levels around the different roles a user may take on when interacting with an agent: operator, collaborator, consultant, approver, and observer.

As a potential application of our framework, we motivate and propose AI autonomy certificates as a new governance mechanism for agents. These certificates specify the maximum autonomy level at which an agent is permitted to operate and have applications in assessing autonomy risks and designing multi-agent systems. Finally, to address the open question of autonomy evaluation, we propose assisted evaluations as a setup that allows autonomy to be evaluated alongside, but distinctly from, capabilities.

As AI agents increasingly make their way out of research labs and into the real world, careful design of agents’ interactive behaviors is needed to fully realize the technology’s potential and bolster, not erode, human agency.