Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach

Conversational AI agents are commonly applied within single-user, turn-taking scenarios. The interaction mechanics of these scenarios are trivial: when the user enters a message, the AI agent produces a response. However, the interaction dynamics are more complex within group settings. How should an agent behave in these settings? We report on two experiments aimed at uncovering users’ experiences of an AI agent’s participation within a group, in the context of group ideation (brainstorming). In the first study, participants benefited from and preferred having the AI agent in the group, but participants disliked when the agent seemed to dominate the conversation and they desired various controls over its interactive behaviors. In the second study, we created functional controls over the agent’s behavior, operable by group members, to validate their utility and probe for additional requirements. Integrating our findings across both studies, we developed a taxonomy of controls for when, what, and where a conversational AI agent in a group should respond, who can control its behavior, and how those controls are specified and implemented. Our taxonomy is intended to aid AI creators to think through important considerations in the design of mixed-initiative conversational agents.

Introduction. The widespread availability of conversational agents, particularly those based on large language models (LLMs), has affected how people complete tasks. It is increasingly common for people to ask an agent for assistance with a writing task [34], a coding task [52, 64, 83], or a summarization task [2, 26]. Often, these tasks are implemented through conversational user interfaces in which the interaction mechanics are trivial: the (human) user makes a statement or request to the AI agent and the AI agent generates a reply via LLM inference using the conversational history as context.

In contrast to the one-on-one nature of these human-AI interactions, a large number of professional work projects occur in groups, where asynchronous and real-time group discussions take place [9]. AI agent participation in such discussions could help provide additional ideas and resources that enrich both the work processes and outcomes [65, 79].

However, in group conversations, it is not straightforward to determine when and how an AI agent should contribute.

For example, it might be inappropriate for the agent to respond to every utterance made by every member of the group in a lively discussion. Group interaction by an agent requires a more flexible strategy involving judgement of what content should be shared when.

To understand the challenges of designing an agent that “fits in” to a group setting, we built an LLM-based conversational agent prototype called “Koala.” Koala is an AI-based group discussion participant, situated in Slack1 and implemented as a bot application. We conducted two user studies to understand Koala’s impact on the dynamics of an ideation activity and to explore the design space for its interactive behaviors.

In our first user study, we established an initial understanding of the value of including an AI agent within a group brainstorming session as perceived by study participants, along with human-centered requirements for the agent’s behavior. We engaged small groups in real-time brainstorming conversations (akin to [5, 67, 75]), in Slack channels configured with different levels of AI participation: (1) without Koala, (2) with a reactive variant of Koala that only responded when directly addressed, and (3) with a proactive variant of Koala that made its own determinations of when to contribute. Participants preferred completing the exercise with, rather than without, Koala, and they reacted positively to the ideas it generated and the way it could help keep the conversation moving. However, participants raised significant concerns about feeling overwhelmed and interrupted by the proactive variant of Koala and expressed numerous ways in which they desired to control its behavior.

Related work. We outline three areas relevant to our study of AI agent participation in group settings: research on mixed-initiative interfaces and frameworks for human-AI interaction, the act of brainstorming as a collaborative activity, and guidelines for the design of conversational agents.

2.1 Mixed-initiative interfaces and frameworks for human-AI interaction One of the earliest technical accounts of joint work between human and computing technologies was what has become known as the Fitts Allocation, published in 1951 [15]. Fitts proposed that certain types of work should be assigned to either a human or a machine, according to a set of rules about which party was best suited to perform which tasks.

Newer frameworks were developed by Sheridan [68] and Parasuraman et al. [59], but these contained the same basic assumption that human initiative and agency traded off with technological initiative and agency.

Recent research on mixed-initiative interactions [23] and mixed-initiative creative interfaces [12, 73], along with modern human-AI interaction frameworks [51, 69] suggest that initiative between humans and intelligent systems does not lie on a single trade-off axis; rather, they comprise a two-dimensional space in which both parties can independently act with lower or higher levels of initiative. Muller and Weisz [51] further demonstrate how the initiative of each actor may dynamically change during the course of the interaction.

One limitation of these frameworks is that, although they can describe that an AI agent can act with lower or higher levels of initiative, they do not help designers and practitioners determine how that agent ought to behave in any given situation. Recent work has examined the design space of proactive interactions, such as by passively recommending visualizations in a data science context [76] or by determining when to interrupt a user with a notification or informational message [43, 63]. As Kraus et al. [31] note, proactivity can be a “double-edged sword” that must be configured properly to maintain users’ preferences and trust [30, 45]. Without such configuration, users may resist agentprovided information, such as help services [20]. Jain et al. [25] used participatory methods to create design patterns for a proactive auto-response messaging agent. Zargham et al. [88] used a different approach – visual storyboards – to learn users’ preferred patterns for what they called the “proactivity dilemma” in design. Gammelgård-Larsen et al. [17] built prototypes to compare users’ responses to intermittent vs. continuous vs. proactive agent behaviors in a music recommendation application. Liao et al. [35] summarized portions of these learnings in a full-day SIGIR tutorial.

Method. To support our research goals, we developed Koala, a multi-party conversational agent designed to collaborate with people in group ideation tasks. We built Koala as a Slack application [72] to enable it to participate in group chat spaces; we specifically chose Slack because it is the primary chat application used within our organization and its API enabled us to develop an AI agent that could participate in group conversations. From a user perspective, “Koala” was just another participant within a Slack channel [32, 66], with an “APP” tag to indicate that it was not a person [70].

We used the Llama 2 model [78] to drive Koala’s conversational capability as it was a state-of-the-art model2 that provided the highest-quality responses in our internal testing.

Manuscript submitted to ACM 6 Houde et al.

An important design dimension of Koala is determining when it should make a conversational utterance. Other Slack bots, such as Anthropic’s Claude3, require users to explicitly mention the bot’s name to elicit a response from it (i.e.

“@”-ing the bot). Given our desire to experiment with proactive ways of participating, we developed two configurable variants of Koala: a reactive variant that only responds when directly addressed (by either “@Koala” or “Koala”), and a proactive variant that responds when it determines that it has something valuable to say. Koala can contribute to the conversation in the same manner as other human users, by sending textual messages as well as emoji-based reactions to existing messages. We show a screenshot of Koala in Figure 1.

3.1 Determining when to speak: Koala’s proactive behaviors Fig. 2. Koala operational logic. When users post a message in a Slack (A), the post triggers an event that is handled by the Koala backend (B) where control logic determines whether Koala should either immediately pass on replying or hand off for further evaluation in the autonomy control logic (C) where the LLM (D) generates a response that is further evaluated for potential posting in the channel.

Conversational AI agents typically engage in one-on-one conversations with human users, where each message from the user receives a response from the agent. In a group chat where multiple people are talking to (and sometimes on top of) one another, the decision of when a proactive agent should respond to any particular user message becomes non-trivial; some mechanism is needed to determine when the agent should respond to a user’s message [18, 21, 57].

In developing the proactive variant of Koala, we knew that writing imperative decision logic to determine what constitutes a valuable contribution would prove impossible; therefore, we relied on the underlying LLM to not only produce conversational responses in response to users’ messages, but also to score those responses for the extent to which they made a valuable contribution to the conversation. We outline this process in Figure 2; at a high level, Koala is instructed to not reply to messages where it is clearly not the intended recipient, and to reply to messages when it estimates that the response it generated makes a valuable contribution to the conversation. We note that our proactive variant of Koala represents a limited form of proactivity as it’s process is triggered by the receipt of a message from a human user; a truly proactive variant of Koala would also be able to send messages to the group even when the human participants remain silent [84]. We provide a complete listing of the prompts used by Koala to generate responses to members of the group and score them for their value in Appendix A.

Manuscript submitted to ACM Controlling AI Agent Participation in Group Conversations: A Human-Centered Approach 7 During Koala’s development, we noticed that the underlying LLM was unreliable in identifying the intended target for a chat utterance.

Discussion. 7.1 Rethinking proactivity In Study 1, we discovered that proactive behaviors were valuable but challenging to implement. Participants reported that Koala’s contributions were sometimes overwhelming and distorted their discussion. Some participants directly asked Koala to reduce its volume of brainstorm posts, indicating the need for users to have the ability to control the behaviors of the agents with whom they interact.

Previous framings of mixed-initiative interaction have tended to treat proactive and reactive behaviors as binary options. McComb et al. [42] provide a clear example of this contrast, writing, “For example, Alexa or Siri react to queries or instructions. Although digital assistants may leverage data from other users to continuously improve in the background, tasks for a user are performed only when prompted by a user. Fully autonomous vehicles utilize proactive AI, conducting tasks such as braking or turning without user input.” [42, p.1907, italics added] Using examples from a diverse range of consumer products, Shneiderman [69] took a different view and proposed a framing in which human control and AI control (“automation”) could be treated as separate, independent dimensions.

Shneiderman’s view appears to treat proactive behaviors as a matter of degrees, rather than as a binary construct.

However, Shneiderman also appears to treat proactivity as a fixed and static attribute of each of the products in his analysis. Building on Shneiderman’s analysis, Muller and Weisz [51] showed that the degree of relative initiative (i.e., human control and/vs. automation) might change during the course of use of an application. In Study 1, we observed participants attempt to control such initiative in real time by trying to reduce the proactive Koala variant’s volume of posts and to increase reactive Koala’s volume of posts. These attempts suggest, contrary to McComb et al. [42]’s framing, that proactivity is not a simple binary property of an AI agent, and that, contrary to Shneiderman [69]’s framing, proactivity may need to be modified in the midst of usage.

Many researchers have cautioned that proactive behaviors could be rejected by users if they are not properly configured [20, 30, 31, 45]. Akin to Gammelgård-Larsen et al. [17], we built Koala II as a prototype to explore different configurations of proactive behaviors. In contrast to their work, in which they provided users with three, fixed configurations (intermittent, continuous, and proactive recommendations), we examined a set of controls in which participants could create their own configurations for the level of participation they desired from Koala II. We note that although much research has focused on providing controls for users to control the outcomes when working with a generative AI application (e.g. [8, 10, 39, 40]), not much work has examined the use of controls to shape the behavior of an LLM-based AI agent.

7.2 Taxonomy for the design of proactive AI agent behaviors Our work highlights how proactive behaviors are comprised of a set of complex, dimensionalized attributes. In Study 1, we captured a portion of this design space; in Study 2, we validated and expanded this design space. In this section, we describe our taxonomy for the design of proactive AI agent behaviors that identifies key aspects of control over an agent’s interactive behaviors and ways to control them (Figure 7). We conclude this section by recontextualizing our taxonomy with prior literature on mixed-initiative interaction frameworks.