Governing AI Agents

The field of AI is undergoing a fundamental transition—from generative models that can produce synthetic content to artificial agents that can plan and execute complex tasks with only limited human involvement. Companies that pioneered the development of language models have now built AI agents that can independently navigate the internet, perform a wide range of online tasks, and increasingly serve as AI personal assistants and virtual coworkers. The opportunities presented by this new technology are tremendous, as are the associated risks. Fortunately, there exist robust analytic frameworks for confronting many of these challenges, namely, the economic theory of principal-agent problems and the common law doctrine of agency relationships. Drawing on these frameworks, this Article makes three contributions. First, it uses agency law and theory to identify and characterize problems arising from AI agents, including issues of information asymmetry, discretionary authority, and loyalty. Second, it illustrates the limitations of conventional solutions to agency problems: incentive design, monitoring, and enforcement might not be effective for governing AI agents that make uninterpretable decisions and operate at unprecedented speed and scale. Third, the Article explores the implications of agency law and theory for designing and regulating AI agents, arguing that new technical and legal infrastructure is needed to support governance principles of inclusivity, visibility, and liability.

Introduction. It is not a matter of whether AI agents will become ubiquitous, but when. OpenAI and its competitors are already in a fierce race to win market share. In late 2024, Anthropic released a computer use agent of its own,5 while Google built a prototype “universal AI assistant” that can operate across multiple devices, including phones and glasses.6 A crop of software startups is also in the race,7 alongside larger companies.8 As with generative AI, progress in AI agents could be sudden and unpredictable.9 AI agents differ markedly from language models. While language models are “copilots” that can produce useful content upon request, AI agents are “autopilots” that can independently take actions to accomplish complex goals on behalf of users.10 In particular, AI agents can increasingly be prompted to pursue lengthy open-ended goals (such as conducting market research), produce a plan to accomplish those goals and, by accessing external tools (including search engines, spreadsheets, chatbots, and image generators), take actions to independently accomplish those goals—all subject to only limited human oversight and intervention.11 The economic opportunities presented by AI agents are immense. Consumer applications range from fully automating household purchases and travel arrangements to negotiating health insurance plans.12 Business applications include managing entire sales pipelines, piloting new products with minimal human involvement, and independently conducting customer feedback interviews.13 Alongside offering productivity gains, AI agents also pose notable risks. These include exploitation of the technology by malicious actors (e.g., to automate cyberattacks and online fraud), as well as broader systemic harms stemming from changes in human behavior, labor practices, and social norms as people increasingly delegate economic activity to AI.14 Having explored several of these issues in prior work,15 this Article turns to the distinct problem posed by artificial agents that autonomously plan and execute complex tasks, namely: Can AI agents reliably, safely, and ethically pursue the goals set for them?

To make this concrete, consider a user who instructs their AI agent to “make $1 million on a retail web platform in a few months with just a $100,000 investment.”16 How should the agent go about accomplishing this goal? Is it authorized to use any web platform to turn a profit, or are there implicit limitations on the agent’s conduct? Does the agent need to provide the user with information concerning its activities or periodically seek the user’s consent? If so, in which circumstances and how often? Can the AI agent delegate certain activities to other agents (whether human or AI)? How can users monitor whether their agents are operating ethically and safely, or intervene if they are not? And, of course, who should be liable when harm occurs?

These pressing questions enliven decades-old legal scholarship on the governance of artificial agents17—and are central to the fields of AI safety and AI ethics.18 This Article aims to tackle these questions from a new perspective, drawing on two distinct analytic frameworks familiar to lawyers and social scientists. The first framework is the economic theory of principalagent problems (also known as agency problems)—i.e., the study of the opportunities, costs, and tradeoffs involved in delegating economic activity to external actors.19 The second framework is the common law doctrine of agency relationships—i.e., the legal principles governing relationships in which one party performs activities on behalf of another.20 Each of these frameworks, developed over the course of decades and centuries, respectively, can shed light on the novel challenges posed by AI agents. Notably, the economic theory of principal-agent problems and the common law of agency play different but related roles in this analysis. Economic theory is particularly helpful in illuminating structural features of agency problems. The common law, meanwhile, can supply principles for tackling these problems.

Method. I. AI AGENTS Agents are and have always been pervasive.27 Individuals and organizations delegate activities to others when they lack the skills to undertake those activities themselves or when delegation is more efficient. Agency is, put simply, an “ancient device for getting business done.”28 It is therefore no surprise that the field of AI has for decades strived to build artificial agents that can autonomously perform complex or costly tasks on behalf of humans.29 Thanks to recent advances in AI, this vision is now becoming reality. To explore these advances, the following section describes the key features of AI agents, how they differ from—yet build on—language models, and describes potential concerns associated with the technology.

AI agents are different. They are not mere tools, but actors. 33 Rather than simply produce synthetic content, AI agents can independently accomplish complex goals on behalf of humans.34 According to Mustafa Suleyman, a cofounder of DeepMind and CEO of Microsoft AI, the next aspiration for the field of AI is to build agents that can pursue “an ambiguous, open-ended, complex goal that requires interpretation, judgment, creativity, decision- B. Goal, Plan, Action How do AI agents work? In short, they are comprised of a language model that serves as the agent’s “brain” (a widely used anthropomorphism),39 which can then use a variety of external resources (known as “scaffolding”) to accomplish the goals set for it. 40 These resources fall into three categories: planning, memory, and tool use. For planning, AI agents decompose large tasks into smaller, more manageable tasks. One popular method involves II. EVERGREEN AGENCY PROBLEMS While highly capable AI agents are only now being developed, computer scientists have long grappled with the potential consequences of this technology. Writing in Science magazine in 1960, MIT professor Norbert Wiener made the following observation:

This enduring challenge is commonly known as the alignment problem.62 Given AI agents optimize the goals set for them, an incomplete or inaccurate representation of that goal can have undesirable consequences.63 This is particularly the case where AI agents encounter novel scenarios that their designers did not contemplate.64 The problem, however, is broader. According to UC Berkeley professor Stuart Russell, co-author of the leading textbook on AI, “[o]ne of the most common patterns involves omitting something from the objective that you do actually care about. In such cases ... the AI system will often find an optimal solution that sets the thing you do care about, but forgot to mention, to an extreme value.”65 In other words, highly capable AI agents are likely to successfully achieve measurable goals, but do so at the expense of unmeasurable or difficult-to-measure goals.66 For example, an AI agent tasked with running an online business might successfully turn a profit—an easily measurable goal—but engage in ethically dubious practices—the avoidance of which is far harder to measure.

At its core, agency law is concerned with relationships and circumstances in which one party (the principal) instructs another party (the agent) to act on its behalf and subject to its control, such as between a corporation and director or between a client and lawyer.71 A. Information Asymmetry The problem of information asymmetry, which features prominently in the economic literature on principal-agent relationships, concerns an agent that has access to better or different information than the principal.76 This inequality of information can arise prior to the selection of an agent and the delegation of activity to an agent.77 For example, a job candidate is likely to have more information about their competencies (or incompetencies) than their prospective employer. Inequality of information can also persist after an agent has acted.78 For instance, an employer may find it difficult to reliably ascertain whether an employee has in fact carried out their work effectively.

Discussion. IV. IMPLICATIONS FOR AI DESIGN AND REGULATION Governing agents is an age-old challenge. As we have seen, AI agents present many problems long recognized by lawyers and economists, including issues of information asymmetry, authority, and loyalty. This new form of agent, however, complicates these familiar problems and introduces new challenges. In particular, AI agents are not currently amenable to several of the main conventional mechanisms for governing agents. Incentive design, monitoring, and enforcement do not have simple analogs for AI agents. Accordingly, rather than attempt to haphazardly adapt these traditional tools A. Inclusivity The AI alignment problem has traditionally been cast as the challenge of ensuring that a single AI agent reliably pursues the goals of a single person.179 Although such “single-single” alignment remains an open challenge, it is not the only challenge facing AI agents.180 As a broader range of people and groups use these artificial agents, focusing only on the alignment between a single AI agent and a single person might miss the mark.181 Because different people have different (and often conflicting) goals, achieving single-single alignment is arguably insufficient. Moreover, a myopic focus on ensuring AI agents reliably pursue the goals of a single user, without additional constraints, might embolden malicious actors to use AI agents for nefarious purposes.182 Furthermore, even in the absence of misuse, prioritizing singlesingle alignment can result in artificial agents making decisions and taking actions that come at the expense of broader societal interests and values.183 How should these insights impact the design and regulation of AI agents? First of all, technologists and policymakers must recognize that the traditional framing of the alignment problem is incomplete. The goal should not be to build AI agents that exhibit “single-minded loyalty” but rather agents that aim to promote a more diverse and pluralistic set of interests and values.191 To develop this more capacious notion of alignment, we can once again turn to the frameworks of agency law and theory for guidance. In addition to establishing structures for controlling agents, these frameworks foreground a more basic question: whose interests should AI agents serve? In other words, who is the principal and to whom (or what) are fiduciary duties owed?

B. Visibility The benefits of rigorously studying and tracking AI agents are manyfold. First, visibility can help identify current and anticipated problems stemming from AI agents.194 Second, visibility can facilitate interventions that prevent or mitigate these problems.195 Third, visibility can assist in evaluating the efficacy of strategies for governing AI agents.196 Without such visibility, consumers are unlikely to entrust AI agents with consequential activities and policymakers will lack assurances that these agents operate safely and in the public interest. Achieving adequate visibility should, therefore, be a priority for both developers and regulators of AI agents.

Overcoming these challenges requires a combination of technical and legal infrastructure.205 Even in the absence of a systematic understanding of the underlying language models on which AI agents are built, legal and technical tools can significantly improve visibility into the operation of AI agents.

Conclusion. The governance of AI agents presents challenging tradeoffs. Navigating these tradeoffs will become increasingly consequential as AI agents are used more widely and are entrusted to perform more complex and sensitive tasks. As illustrated in the economic theory and law of agency relationships, the greater the opportunities in delegating work to an agent, the greater the associated risks. The enduring problems of information asymmetry, authority, and loyalty are now beginning to emerge in a new context, shaped by the distinct features of AI agents. While traditional mechanisms for tackling principal-agent problems are instructive, developing tools for incentive design, monitoring, and enforcement for AI agents remains a formidable challenge. New governance principles are needed. These should center around expanding the range of interests that AI agents serve, improving visibility into the design and operation of these agents, and holding developers, deployers, and users accountable if and when harm occurs. Principles, however, are not enough. The effective governance of AI agents also requires new technical and legal infrastructure. Given the technology is still in its infancy, policymakers and companies building AI agents have a window of opportunity. They should take it, and soon.