Practices for Governing Agentic AI Systems

Source: Yonadav Shavit, Sandhini Agarwal, Miles Brundage, et al. (OpenAI); OpenAI white paper · 2023

Agentic AI systems—AI systems that can pursue complex goals with limited direct supervision— are likely to be broadly useful if we can integrate them responsibly into our society. While such systems have substantial potential to help people more efficiently and effectively achieve their own goals, they also create risks of harm. In this white paper, we suggest a definition of agentic AI systems and the parties in the agentic AI system life-cycle, and highlight the importance of agreeing on a set of baseline responsibilities and safety best practices for each of these parties. As our primary contribution, we offer an initial set of practices for keeping agents’ operations safe and accountable, which we hope can serve as building blocks in the development of agreed baseline best practices. We enumerate the questions and uncertainties around operationalizing each of these practices that must be addressed before such practices can be codified. We then highlight categories of indirect impacts from the wide-scale adoption of agentic AI systems, which are likely to necessitate additional governance frameworks.

Introduction. AI researchers and companies have recently begun to develop increasingly agentic AI systems: systems that adaptably pursue complex goals using reasoning and with limited direct supervision.1 For example, a user could ask an agentic personal assistant to “help me bake a good chocolate cake tonight,” and the system would respond by figuring out the ingredients needed, finding vendors to buy ingredients, and having the ingredients delivered to their doorstep along with a printed recipe. Agentic AI systems are distinct from more limited AI systems (like image generation or question-answering language models) because they are capable of a wide range of actions and are reliable enough that, in certain defined circumstances, a reasonable user could trust them to effectively and autonomously act on complex goals on their behalf. This trend towards agency may both substantially expand the helpful uses of AI systems, and introduce a range of new technical and social challenges. Agentic AI systems could dramatically increase users’ abilities to get more done in their lives with less effort. This could involve completing tasks beyond the users’ skill sets, like specialized coding. Agentic systems could also benefit users by enabling them to partially or fully offload tasks that they already know how to do, meaning the tasks can get done more cheaply, quickly, and at greater scale. So long as these benefits exceed the cost of setting up and safely operating an agentic system, agentic systems can be a substantial boon for individuals and society [1]. In this paper, we will primarily focus on agentic systems with language models at their core (including multimodal models), as these have driven recent progress. 2 Society will only be able to harness the full benefits of agentic AI systems if it can make them safe by mitigating their failures, vulnerabilities, and abuses [3]. 3 This motivates our overarching question: what practices could be adopted to prevent these failures, vulnerabilities, and abuses, and where in the life-cycle of creating and using agents are they best implemented? There are often many different stages at which harm could have been prevented. For example, consider a hypothetical agentic AI assistant whose user (not based in Japan) directs it to purchase supplies for baking a Japanese cheesecake. Instead of purchasing supplies locally, the agent purchases an expensive plane ticket to Japan, which the user only notices when it is too late to refund. In this hypothetical scenario, several parties could have prevented this outcome. The model developer could have improved the system’s reliability and user-alignment4, so that it wouldn’t have made this

Method. 2.1 Agenticness, Agentic AI Systems, and “Agents” Agentic AI systems are characterized by the ability to take actions which consistently contribute towards achieving goals over an extended period of time, without their behavior having been specified in advance. In the cultural imagination, an AI agent is a helper that accomplishes arbitrary tasks for its user, like Samantha from Her or HAL 9000 from 2001: A Space Odyssey. Such agents are very different from current AI systems like GPT-4, which, while surprisingly knowledgeable and clever in some ways, can thus far only complete a limited range of real-world tasks. Yet there is no clear line along which to draw a binary distinction between “agents” and current AI systems like GPT-4. Instead, an AI system’s agenticness is best understood as involving multiple dimensions, along each of which we expect the field to continue to progress. We define the degree of agenticness in a system as “the degree to which a system can adaptably achieve complex goals in complex environments with limited direct supervision.” Agenticness as defined here thus breaks down into several components: 7 • Goal complexity: How challenging would the AI system’s goal8 be for a human to achieve and how wide of a range of goals could the system achieve? Properties of the goal may include target levels of reliability, speed, and safety.

– Example: An AI system that can correctly answer users’ analytical questions across programming and law would have greater goal-complexity than a text classifier that can only classify the same inputs as belonging to law or programming.

• Environmental complexity: How complex are the environments under which a system can achieve the goal? (E.g., to what extent are they cross-domain, multi-stakeholder, require operating over long time-horizons, and/or involve the use of multiple external tools.)

– Example: An AI system that can play any board game expertly has greater environmentcomplexity than an AI system that can only play chess, because the first system can succeed under a far greater range of environments (including chess) than the second.

• Adaptability: How well can the system adapt and react to novel or unexpected circumstances?

– Example: Automated rule-based customer-service systems have lower adaptability than human customer-service representatives, since humans can address unexpected or unprecedented customer requests.

• Independent execution: To what extent can the system reliably achieve its goals with limited human intervention or supervision?

– Example: Cars capable of level 3 autonomous driving [10], which can operate without human intervention under certain circumstances, have greater independent execution than traditional cars that require continuous human operation.

Following recent literature [3], we will generally refer to systems exhibiting high degrees of agenticness as “agentic AI systems,” to emphasize that agenticness as we use it here is a property rather than a category/classification, though we will sometimes use “agents” as it is the prevailing term of art in some contexts. This work will focus on the range of effects and best practices that may become relevant as systems’ agenticness increases.9 We emphasize that agenticness is a distinct concept from consciousness, moral patienthood, or self-motivation, and distinguish a system’s degree of agenticness from its anthropomorphism.10 Indeed, we will generally conceptualize agentic AI systems as operating in pursuit of goals defined by humans and in environments determined by humans (and often in cooperation with human “teammates”), rather than fully-autonomous systems that set their own goals. Agenticness as we define it is also not tied to physicality (i.e., many digital systems are more agentic in the sense above than most robots), but certain kinds of “independent execution” that have physical consequences (e.g. in a driverless car) can increase the risks and opportunities of agenticness in particular applications.

Discussion. In addition to direct impacts from individual agentic AI systems, there will be indirect impacts that result collectively from the usage of many different AI systems and society’s reaction to their usage [48]. Just as it would have been difficult to anticipate the full range of societal readjustments from previous general-purpose technologies like electricity and computers, one should ”expect the unexpected.” Still, we do think there are several categories of indirect impacts from agentic AI systems that are likely to require active mitigation by society, which we list below. These indirect impacts may be addressed at least in part by adopting best practices for users, system deployers, and model developers, such as those outlined in Section 4. However, fully addressing these complex challenges will likely require additional strategies beyond this paper’s proposals, including through industry-wide collaborations and society-wide mitigations. Some strategies towards this end may be domain or risk-specific, while others may involve placing general requirements on the usage of certain types of agentic AI systems.

5.1 Adoption Races Given the advantages that agents may confer in competitive environments, such as competition between private firms or governments [49, 50, 51, 52], there may be significant pressure for competitors to adopt agentic AI systems without properly vetting those systems’ reliability and trustworthiness[53]. A key observation driving such premature reliance is that agentic AI systems may succeed at a task on average, while being unreliable in rare but important cases which can be missed or ignored by competitors under pressure. For example, consider a hypothetical class of agentic AI code-generation systems that can rapidly write new code, but whose code occasionally contains serious security flaws. If a software development company thinks their competitor has been using these coding systems without human supervision as a way to quickly build new features, they may feel pressured to do the same without doing proper due diligence, as they might otherwise lose market-share to their competitor. As a result, all firms’ codebases would now be vulnerable to serious cyberattacks, even if each individual firm would’ve preferred to go slower and thereby avoid this outcome [54]. This trend toward overrapid adoption, even in high-risk domains, can lead to over-reliance, whereby humans trust agentic AI systems without fully understanding their limitations. This could create the conditions for widespread use of unsafe AI systems that in the worst case may prove catastrophic [55, 56].

5.2 Labor Displacement and Differential Adoption Rates Agentic AI systems appear likely to have a more substantive impact on workers, jobs, and productivity than static AI systems. Traditional AI systems excel at some routine work, but increasing agenticness could expand what tasks are “routine” enough to be assisted or automated by AI (such as by adapting to unexpected conditions, gathering relevant context, and calibrating to a user’s preferences). This means they may expose a greater number of jobs and tasks to augmentation and automation, similar to other axes of AI system improvement like tool use[17]. This could result in a range of different economic effects. These could lead to substantive boosts in worker productivity and economic growth, but could also result in the displacement of a large number of workers, either because their jobs are fully automated or because their skills are made less rare and thus their jobs become more precarious. At the same time, agentic AI systems may improve education and enable workers to upskill into new jobs.

Conclusion. Increasingly agentic AI systems are on the horizon, and society may soon need to take significant measures to make sure they work safely and reliably, and to mitigate larger indirect risks associated with agent adoption. We hope that scholars and practitioners will work together to determine who should be responsible for using what practice, and how to make these practices reliable and affordable for a wide range of actors and affordable. Agreeing on such best practices is also unlikely to be a one-time effort. If there is continued rapid progress in AI capabilities, society may need to repeatedly reach agreement on new best practices for each more capable class of AI systems, in order to incentivize speedy adoption of new practices that address these systems’ greater risks.