Visibility into AI Agents

Increased delegation of commercial, scientific, governmental, and personal activities to AI agents—systems capable of pursuing complex goals with limited supervision—may exacerbate existing societal risks and introduce new risks. Understanding and mitigating these risks involves critically evaluating existing governance structures, revising and adapting these structures where needed, and ensuring accountability of key stakeholders. Information about where, why, how, and by whom certain AI agents are used, which we refer to as visibility, is critical to these objectives. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. For each, we outline potential implementations that vary in intrusiveness and informativeness. We analyze how the measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Further work into understanding the measures and mitigating their negative impacts can help to build a foundation for the governance of AI agents.

Introduction. Many AI developers are creating systems with greater autonomy, access to external tools or services, and an increased ability to reliably adapt, plan, and act open-endedly over long time-horizons to achieve goals [35, 97, 106, 133, 137, 146, 158]. We will say that such systems possess relatively high degrees of agency and will refer to them as (AI) agents or agentic systems [35, 104, 147]. Systems with relatively low degrees of agency are those that only aid human decision-making or produce outputs without acting in the world, such as image classifiers or text-to-image models. Examples of agents could include reinforcement learning systems [132, 160] that interact extensively with the real world1 or more capable versions of language models with tool or service access that could, for example, plan and book a holiday or send an email on a user’s behalf [27, 122, 137, 144]. Current AI agents sometimes struggle to perform even simple tasks [98, 106, 108, 130, 163, 164], but given increasing investments in AI research [61], scaling laws [15, 83, 94], pressures to develop autonomous capabilities for military use [86, 102, 142], economic applications [35], and scientific prestige [35, 67], we should not discount continued improvements in capabilities [26]. Indeed, a core goal of the AI field since its inception has been to build agents [138, 159]. As AI agents improve in capabilities, speed, and cost,2 it may be easier and more competitive to delegate tasks currently done by humans to AI agents instead. The development and deployment of agents has surged recently [36, 124, 133, 150, 176] and could lead to the ubiquitous deployment of agents in commercial, scientific, governmental, and personal activities. Since such deployment may exacerbate existing risks and introduce new ones [35, 147], it is imperative to understand how to govern AI agents.

Rather than provide an exhaustive taxonomy of risks from AI agents,3 we highlight certain agent-specific risks. In comparison to risks from other AI systems, these risks focus on the potential for agents to remove humans from the loop [35, 101]. Without a human in the loop, agents may take multiple consequential actions in rapid succession and bring about significant impacts before a human notices. The ability to remove humans from the loop also means that an agent’s task performance is less limited by the expertise of its user, compared to a situation where user must guide an AI system’s actions or take actions themself.

1.1.1 Malicious Use. AI agents could be a large impact multiplier for individuals or coordinated groups who wish to cause harm [147]. Existing AI systems have already assisted in malicious use, including voice cloning scams [168] and fake news generation [167]. However, more capable AI agents could automate end-to-end pipelines for complex tasks that currently require substantial human expertise and time. For untrained individuals, such agents could drastically increase the accessibility of engaging in severely harmful activities because no human in the loop would be required. For example, there is interest in building agents to execute scientific research, comprising autonomous planning and execution of scientific experiments [22, 27]. If such agents were to become as capable as human scientists, they might enable or accelerate the design and development of harmful tools (e.g., biological [140, 155], chemical [22, 27, 162]) for groups that currently lack the expertise for such production. Extremely persuasive AI agents may also enable and enhance influence campaigns [10, 78, 96]. Understanding the extent to which agents will facilitate malicious use requires information about how they are used and how they interact with external systems [173]. Moreover, when malicious users do cause harm with AI agents, regulatory enforcers will need measures to identify the users and hold them accountable.

1.1.2 Overreliance and Disempowerment. Overreliance on AI agents to automate complex, high-stakes tasks could lead to severe consequences. Humans can already rely on certain automated systems more than is warranted [47, 59, 60]. More capable agents may enable automation of an increasing array of complex and useful tasks. Users—including both individuals and institutions—may rely on agents even in high-stakes situations, such as interfacing with the financial or legal systems, because human alternatives (e.g., hiring a lawyer) may become relatively slower and more expensive. At the same time, these agents may malfunction for a variety of reasons, including design flaws [117, 130, 181] or adversarial attack [12, 175, 180]. Malfunction may not be immediately apparent, especially if users lack the requisite expertise or domain knowledge. Stopping the agent may be difficult if doing so would lead to cascading failures or a competitive disadvantage for the user [147].

Method. In this paper, we assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logs. For each, we outline potential implementations that vary in intrusiveness of data collection and informativeness of the data. We analyze how our measures apply across a spectrum of centralized through decentralized deployment contexts, accounting for various actors in the supply chain including hardware and software service providers. Finally, we discuss the implications of our measures for privacy and concentration of power. Rather than advocating for immediate implementation of these measures, we emphasize the need for further understanding them and how to mitigate their negative impacts. The measures extend existing work in deployment visibility to better account for the risks of AI agents. Agent identifiers, which indicate whether and which AI agents are involved in interactions, generalize watermarks [105, 170] because they apply to all of an agent’s outputs, including the use of external tools and services, not just text, image, or audio outputs. This generalization is crucial for improving visibility if agents increasingly substitute for human actions. For real-time monitoring and activity logging, we assess practices that extend existing schemes [80, 91, 92, 111, 151] so as to better track complex interactions between multiple agents [45, 143], an agent’s interaction with external tools or services, and delayed and diffuse effects of an agent’s actions.

Besides the definitions here, we also define each term when we use it for the first time in the main body. We illustrate the most common terms in Figure 1. Agency is the degree to which an AI system acts directly in the world to achieve long-horizon goals, with little human intervention or specification of how to do so. An (AI) agent is a system with a relatively high degree of agency; we consider systems that mainly predict without acting in the world, such as image classifiers, to have relatively low degrees of agency. Examples of agents include reinforcement learning systems [132, 160] that interact extensively with the real world6 or more capable versions of language models with tool access [27, 122, 137, 144]. We do not consider existing foundation models themselves to be agents. Our definitions compress the characterization of agency in Chan et al. [35], which points to four axes: the degree to which the system’s behaviour is specified, the degree to which the system’s behaviour is goaldirected, the degree to which the system has a direct impact in the world, and the degree to which the system can achieve goals over long time-horizons. For the purposes of this paper, we use agent and agentic system interchangeably. Scaffolding is any method that structures the calls to an AI system so as to facilitate the pursuit of goals [36, 133, 176]. Scaffolding may include additional prompts, memory systems, access to external tools, and planning mechanisms [169]. For example, AutoGPT [133] has a language model accept a high-level goal and sequentially produce (reasoning, plan, criticism of the plan, action) tuples so as to achieve the goal. Scaffolding can make an AI system, such as a foundation model, more agentic. The term developer(s) refers to the actor(s) involved in the construction of an AI system. While the developers of a system include those who trained the underlying machine-learning model, developers could also include those who build other components of the complete system, such as the scaffolding [133, 176]. The user is the human individual or group that interacts with and provides instructions to an AI system. The deployer is the entity that operates an AI system and serves it to users. The deployer may not be the same as the developer(s). For example, Microsoft deploys OpenAI’s systems into its products [171], but did not develop GPT-4. A deployer may provide access to an agent in one of two ways. First, the deployer may serve a foundation model which users may combine with other components to make a more7 agentic system.

Discussion. Privacy considerations may conflict with obtaining detailed information about agent activity. Language model deployers are increasingly offering customers, particularly business customers, privacy assurances around data collection and use. Measures to reassure customers about confidentiality include:

• Language model APIs with no logging of inputs or outputs, and the ability to turn off safety filters and moderation classifiers [37]. • Guarantees that customer data, including system outputs, will not be used to train any AI system and will be kept in a customer’s cloud instance [38]. • The ability to delete logs kept by the provider after a certain amount of time [37]. Additionally, existing data protection laws, such as GDPR, impose further restrictions. Agent cards may contain identifying information about users. Agent logs may be considered personal data, such as when agents are given access to a filesystem containing personally identifiable information [64]. In general, if agents substitute for humans in a wide variety of activities, information about those agents might be tantamount to information about the users of those agents. Indeed, agent activities may be easier to monitor than human activities because deployers are a central intermediary. Governments or deployers may thus abuse their power to carry out excessive or unjustified surveillance of personal activities [70, 75]. These considerations justify limiting data collection in accordance with the risk of the agent’s activities or domain of deployment. Another potential mitigation may be decentralized data custody schemes or data trusts [52, 53] whereby users or accountable representatives would make decisions about data usage. Modulating the degree of access to collected information can also help to mitigate privacy concerns. Access can vary with respect to granularity, the amount of detail contained in the records, and quantity, the number of records that a party is allowed to access. With respect to granularity, information can be aggregated, de-identified, or identifiable. Aggregated information involves summary statistics but not individual records or logs; differentially private [20] computations of summary statistics may help to preserve the privacy of individual records. Records and logs can be de-identified with respect to individual users or identifiable. With respect to quantity, a party can have full access to all records, access based on approved search queries or filters, or access upon-request to pre-specified records to which they must provide a compelling reason for access. The granularity and quantity of access should be the minimum necessary for the accessing party to achieve its (legitimate) objectives. When investigations pertain to specific users, identifiable information could be made available upon request given a showing of compelling need and/or after approval from a third-party adjudicator. Regulators may need logs containing identifiable information in some cases, such as oversight of certain high-risk or high-volume activities. For example, for traders transacting above a specified threshold, CFTC collects identifiable personal information to enable aggregation of data across different accounts and brokers [33].

Extending visibility measures to decentralized deployments has serious implications for privacy and concentration of power. Compute providers that surveil deployments may be able to infer sensitive information about users. Given that a handful of compute providers dominates the market [134], monitoring users of those providers would equivalent to monitoring much of society.

Conclusion. Visibility facilitates the governance of increasingly agentic systems. We assessed three mechanisms for visibility: agent identifiers, real-time monitoring, and activity logs. Agent identifiers indicate whether and which agents are involved in an interaction. To aid accountability and incident investigation, an agent card containing additional information about the agent can be attached to an agent identifier. Real-time monitoring aims to flag problematic agent behaviour as it happens. Activity logs record certain inputs and outputs of agents so as to enable in-depth, post-hoc analysis of behaviour. We examined how to extend the visibility measures to decentralized deployments of agents, in particular through using compute providers and tools and services providers to obtain visibility. Finally, we analyzed the implications of the visibility measures on privacy and concentration of power. Rather than advocating for immediate implementation of these measures, further understanding of the measures and how to mitigate their negative impacts is required. Such understanding can help to build a foundation for the governance of AI agents. Visibility informs actions to manage risks from the deployment of increasingly agentic systems, but is not by itself sufficient. Even with a comprehensive understanding of agent activities, those harmed by them may not have the power to intervene and reduce risks [8]. To best make use of visibility, future work could investigate increasing public influence over AI development and deployment [34, 88, 126, 145], developing a wide range of potential policy levers [9, 56, 147], and implementing infrastructure and practices to prevent or defend against harms [29, 139].