Challenges in Human-Agent Communication

Abstract Remarkable advancements in modern generative foundation models have enabled the development of sophisticated and highly capable autonomous agents that can observe their environment, invoke tools, and communicate with other agents to solve problems. Although such agents can communicate with users through natural language, their complexity and wide-ranging failure modes present novel challenges for human-AI interaction. Building on prior research and informed by a communication grounding perspective, we contribute to the study of human-agent communication by identifying and analyzing twelve key communication challenges that these systems pose. These include challenges in conveying information

Introduction. Artificially intelligent agents are systems that can autonomously perceive and take actions in an environment [61]. While the study of AI agents traces back many decades [61, 70, 82], recent advances in generative foundation models that can output novel text or images based on natural language prompts have paved the way for the widespread development and deployment of a new class of agents that are increasingly sophisticated, powerful, and general purpose. Granted the ability to access the internet,1 connect with other applications through APIs,2 and even generate and execute computer code,3 today’s AI agents can perform actions such as scheduling meetings, booking flights,4 ordering food,5 or purchasing groceries, taking actions that impact both the digital and physical realms. Ongoing developments in multi-agent architectures are further expanding the capabilities and use cases of agents [80, 82, 22]. Along with agents’ greater capacity to take actions in the open world and to complete goals on behalf of users comes a wider range of potential failure modes and associated costs [68, 88, 24]. In fact modern applications are extending agent-centric activities into high-stakes scenarios. For example, an agent that can shop on a user’s behalf can spend money in unintended ways or inadvertently leak the user’s address, credit card number, or other sensitive information. An agent that can execute computer code can corrupt files, alter important settings, overwrite family photos or work assignments, and take actions that jeopardize security. Without a proper understanding of an agent’s capabilities and limitations and the ability to verify its actions, a user may over-rely on an agent, leading to the user requesting that the agent perform a task that it is incapable of completing. Particularly in situations where failures are costly or likely to occur, it is critical to build agents that allow users to clearly express their goals, preferences, and constraints to the agent and to form an accurate mental model of how the agent will behave. Users should also be able to monitor the agent’s behavior and effectively guide the agent with feedback and corrections as needed. Put another way, to enable effective collaboration with users, agents and systems of multiple agents must be designed to support transparency and control. The key to enabling transparency and control is effective two-way communication aimed at establishing common ground about the user’s goals (e.g., as represented by the content of the user’s request) and about the process the agent intends to take to achieve these goals [13, 65]. Achieving common ground is an activity that begins with the user’s first introduction to the agent and continues throughout and after usage. Communication between users and agents can help a user to determine when and how much to rely on the agent and, through the iterative nature of dialog about intentions, capabilities, and activities, can help the user to identify and correct misconceptions before irrevocable actions have been taken.

However, communication that establishes common ground can be difficult. Even with smallscale or special-purpose models, it can be difficult to characterize an AI system’s abilities in a way that users understand [18, 45, 76, 43, 53]. Generative foundation models have characteristics that make transparent disclosure of their operation particularly challenging, including the wide and constantly evolving range of tasks they can perform at different levels of competency, their massive and opaque architectures, sensitivity to prompt and context, the stochasticity of their output, and the diversity of their user bases who may require different levels of detail [44]. The challenge of understanding agents’ operation is further exacerbated by the complexity of agentic workflows, in which tasks may be decomposed and carried out over multiple steps— sometimes by multiple interacting agents with different roles, privileges, and access to different information [80, 22]. While the literature on human-AI interaction offers general guidance and sets of principles [29, 1, 26], interacting with systems of one or more autonomous, tool-using AI agents raises new transparency and communication challenges that have yet to be addressed. We present a set of challenges that arise in the process of establishing common ground between human users and AI agents, as summarized in Figure 1. We arrived at these challenges based on our experiences building and experimenting with complex AI agents and multi-agent systems, drawing on the literature on human-AI interaction and collaboration, including prior work on establishing common ground between people and machines. Some of these challenges reflect the difficulty of conveying necessary information from the agent to the user to allow the user to form an appropriate mental model of the agent and monitor its behavior.

Related work. 2 Previous Work We now summarize some of the relevant literature that we build upon, drawing from prior research and results in psychology, cognitive science, AI, and HCI. Researchers in philosophy and psychology have long studied human communication and teamwork. The notion of grounding in communication is a central concept proposed by Clark and Brennan that embodies the collection of mutual knowledge, mutual beliefs, and mutual assumptions that are essential for communication between two people [13]. To effectively ground their communication, the participants must “coordinate both the content and process” of each conversation. The process of achieving a shared mutual belief (common ground) is called grounding. Clark proposed successful grounding as requiring the parallel action of multiple levels of analysis, including the establishment of a communication channel, the exchange of recognized signals across the maintained channel, the interpretation of intentions via decoding meaning in the signals, and the control of the back and forth of an effective conversation with contributions and clarifications being made by dialog participants. Drawing parallels to human communication, Brennan applied grounding theory to humancomputer interaction, where, as in human communication, “people need to be able to seek evidence that they have been understood and to provide evidence about their own intentions” [6]. In this case, the term “common ground” refers to operational alignments rather than cognitive or experiential similarities. Brennan noted that poor feedback or affordance mechanisms often result in users needing to execute laborious checking actions to achieve common ground. The breadth of capabilities (and stochasticity) displayed by generative language models exacerbate these problems, as we discuss. In related work informed by research in psychology on human-human communication, explicit computational grounding machinery was developed and integrated within computational architectures to support human-AI grounding processes as joint activity aimed at achieving mutual understanding [31]. Another effort explored the development of computational analogs of the multiple levels of coordination for grounding proposed by Clark, including inferences at channel, signal, intention, and conversation levels, to establish common ground between a user and a dialog system that could execute actions in the open world. [54]. In later work, definitions of grounding in human-AI collaborations were shifted to an expected utility framework. In the approach, grounding on ideal actions for an agent is guided by uncertainties that an agent has about a user’s goals and the costs and benefits of different actions [54].

Method. 3 Overarching Challenges for Human-Agent Communication Before digging into what specific information must be communicated between a human user and AI agent to establish common ground, we begin with a discussion of four overarching challenges that should be considered across all human-agent communication (challenges X1–X4 in Figure 1). We briefly introduce each here and return to them when discussing the challenges in communicating specific information in Sections 4 and 5.

4 Information Flow from the User to the Agent In this section, we discuss three challenges (U1–U3) related to designing agents to enable users to communicate necessary information. The first two concern the user’s desires: what users want the agent to achieve and preferences for how the agent achieves it. The third challenge pertains to feedback and helping the agent improve over time. While much of the information content covered by U3 overlaps with U1 and U2, the time and context differ, affecting communication. For each issue, we discuss existing research, describe how these difficulties have changed for today’s agents, provide concrete examples, and outline possible solutions.

5 Information Flow from the Agent to the User In this section, we discuss five challenges related to the information flow from the agent to the user. These challenges focus on communicating information about the agent’s capabilities, the agent’s current and planned future actions, whether the user’s goals that have been achieved, and any side effects that have occurred. We provide examples to illustrate each challenge and discuss potential solutions.

Discussion. X1: How should the agent help the user verify its behavior?

Modern agents are powerful but remain imperfect. As a result, when agents get tasked with goals, it is likely that they will make mistakes—especially when these goals are complex or multi-step. Consider an agent tasked with fixing a programming bug in a GitHub repository. This agent can fail in numerous ways. For example, it might fail to understand the issue and its requirements, such as the fact that any solution must be backward compatible; it might develop a plan that implements the fix in the source directory but neglects to update the test cases; it might propose a fix that is “unsafe” because it is vulnerable to exploits or propose a fix that is convoluted and time-consuming to review; it might propose a change that has side effects such as removing unrelated functionality. To establish establish common ground, it is necessary not only to convey pieces of information but to verify that a common understanding has been reached. In the context of human-AI communication, it is not enough, for instance, for a user to state their goals and preferences if these goals and preferences are misunderstood or ignored by the agent. It is, therefore, critical to enable the user to verify their understanding of the agent’s plans and actions as well as the agent’s understanding of their own goals, preferences, requirements, and feedback. Establishing mechanisms for the user to easily verify the agent’s behavior is a necessary condition of effective communication [21] and an important aspect of effective teamwork [39]. Without this ability, the user may not be able to steer agent behavior to avoid costly mistakes or accomplish the goal. Furthermore, they would fail to develop an accurate mental model of the agent’s capabilities or provide feedback to improve the agent’s future performance.

X2: How should the agent convey consistent behavior?

A second overarching challenge for human-AI communication is how to prevent the agent from confusing the human with behavior or outputs that are (or at least appear) inconsistent. Inconsistency can arise for multiple reasons. One is the agent’s behavior’s inherent stochasticity, which stems from two main sources: the probabilistic nature of the underlying foundation model’s outputs, and the complex interaction patterns that emerge during task execution. Even with deterministic foundation model outputs (e.g., a model with temperature set to 0), when agents operate in dynamic environments, their orchestration logic—which governs action selection, inter-agent delegation, and task completion criteria—can produce different sequences of actions across runs as the environment changes in response to agent actions. While stochastic behavior may be desirable in some cases—for instance, if the user would like the agent to generate a variety of different options to choose from—in other cases, it can hinder the user’s ability to create an accurate mental model of the agent. For example, consider an LLM-based agent that can create visualizations based on a high-level description from the user. When invoked with the same input multiple times, the agent might choose different libraries or tools to plot the visualization. Sometimes, it might write code to fetch the data from a hosted service; other times, it might generate that data from its memory. It could also unpredictably apply different visual styles, which would be especially problematic if the user is unaware. Perceived inconsistencies can arise for other reasons, for instance, when the user’s mental model of the world does not match up with the information the agent is acting on.

Conclusion. Generative language models have enabled complex tool-using agents, creating a new breed of AI systems that can execute actions that affect the digital and the physical world—often in sophisticated and sometimes unpredictable ways. These changes have made it increasingly difficult to establish and maintain common ground, a critical goal for effective collaboration. The difficulties of managing the complexity and rapid execution of agent actions, as well as the stochastic nature of their behavior, are new, emerging directly from these technological advancements. The long-standing challenges users face when interacting with AI systems—such as conveying goals, understanding agent competencies, and managing expectations—are increasingly complex with human-agent communication. At the same time, the capabilities of agent-based systems bring heightened stakes of successful communication versus failure. The new complexities coupled with increasingly high stakes have profound implications for transparency and trust. Given these challenges, we issue an urgent call to action for the AI research community: there is critical need for sustained effort to develop new design patterns, guidance, and principles that prioritize enhancing common ground between users and AI agents. As many of the examples illustrate, the importance of developing and maintaining shared understanding cannot be overstated, as it directly affects the effectiveness and safety of AI systems, particularly in environments where the cost of failure is high. Maintaining common ground requires explicit and concerted efforts to create methods and tools that are aimed at establishing and maintaining shared understanding, thereby ensuring agents operate in a way aligned with human needs and expectations. The community must prioritize these challenges by fostering cross-disciplinary research and practical innovation as we advance. Only through these efforts can we build AI systems that are not only powerful but also transparent, trustworthy, appropriately controllable, and capable of maintaining effective communication about their activities with their human counterparts.