Embodied LLM Agents Learn to Cooperate in Organized Teams

Large Language Models (LLMs) have emerged as integral tools for reasoning, planning, and decision-making, drawing upon their extensive world knowledge and proficiency in language-related tasks. LLMs thus hold tremendous potential for natural language interaction within multi-agent systems to foster cooperation. However, LLM agents tend to over-report and comply with any instruction, which may result in information redundancy and confusion in multi-agent cooperation. Inspired by human organizations, this paper introduces a framework that imposes prompt-based organization structures on LLM agents to mitigate these problems. Through a series of experiments with embodied LLM agents and human-agent collaboration, our results highlight the impact of designated leadership on team efficiency, shedding light on the leadership qualities displayed by LLM agents and their spontaneous cooperative behaviors. Further, we harness the potential of LLMs to propose enhanced organizational prompts, via a Criticize-Reflect process, resulting in novel organization structures that reduce communication costs and enhance team efficiency1.

Introduction. Modern intelligent systems, such as autonomous vehicle networks and swarms of drones, often involve complex decision-making processes where multiple agents must collaborate seamlessly to achieve specific objectives [57, 55, 66, 58]. In these systems, communication among the various agents is pivotal, as it dictates the flow of information, coordination of tasks, and overall system performance [69, 14, 10, 8]. Agents in traditional multi-agent systems often have to communicate in pre-specified ways, such as exchanging gradients, sharing data, state observations and actions, etc [20, 27, 10]. The emergence of large language models (LLMs) makes it possible for AI agents to communicate and cooperate using natural language, bringing enormous flexibility and potential for more nuanced and human-understandable interactions [36, 17, 32, 5].

Despite the flexibility of LLMs, integrating them into practical multi-agent systems remains a challenge. While LLMs are trained and finetuned for text generation and instruction-following, they are not necessarily tailored to multi-agent cooperation. Modern LLMs are prone to over-reporting and obeying instructions, as a by-product of RLHF finetuning [2], and they can ignore critical information [28] or be distracted by irrelevant information [46], especially when the context is long (see Figure 1 for examples). While recent studies involving agent-based LLMs have demonstrated they are capable of solving problems through multi-agent collaboration [24, 65, 32], it is worth noting that such collaborations often follow predefined patterns designed using heuristics to channel the behavior of the models productively [24]. Creating systems that support free-flowing interaction between LLMs in a way that could potentially scale to include humans is still an open problem.

This paper investigates the collaborative potential of LLM agents working in teams. Drawing on prior studies in human collaboration from cognitive and economic perspectives, there is potential for organizations to be redesigned to more effectively manage the limited attention span within teams, as suggested by Simon et al. [49], and mitigate individual limitations and enhance overall team performance, as highlighted by Van Zandt [53] and Vélez et al. [54]. Specifically, we study two research questions. First, what role do organizational structures play in multi-LLM-agent systems? Second, how can we optimize these organizational structures to support efficient multiagent coordination? By leveraging AutoGen [60], a generic multi-agent conversation framework, we develop a framework for studying how to best organize embodied LLM agents to communicate and collaborate in physical/simulated non-text environments [65]. Our framework offers the flexibility to prompt and organize LLM agents into various team structures, facilitating versatile inter-agent communication. It also serves as a testbed to empirically evaluate the traditional ideas proposed in the organization theory literature.

Our initial experiments in this setting reveal that uncoordinated LLM agents often send redundant and repetitive messages and interrupt others’ actions, leading to chaos (see Fig. 1 and Appendix E). To remedy these issues, we explore organizational structures, i.e., the dynamics of information exchange, that allow multiple LLM agents to collaborate and complete a common task efficiently.

Related work. As powerful LLMs inherit abundant world knowledge and also general reasoning ability, there are increasing efforts to deploy LLMs as the reasoning core for decision-making to build human-like autonomous agents [50, 74, 15]. This requires observations of the RL environment to be translated into natural language in a way that is easier for LLMs to process. The reasoning of the LLMs also needs to be turned into a viable action for execution. Popular prompting techniques for doing so include ReAct [63] and Reflexion [48]. Other methods that involve fine-tuning the language models have also been explored [16]. In addition, various techniques have been proposed to mitigate the biases and constraints of LLMs, including chain-of-thought reasoning [59], external tools [45, 37], external documents [56] and skill libraries [74].

Multi-agent cooperation has been extensively studied for decades under various topics such as communication efficiency, planning, leadership, and team dynamics using different platforms [30, 44, 43, 40] (see recent surveys for detail [34, 68, 13]). Previous works mainly focused on communication through continuous vectors [8] or discrete symbols [30, 19]. Recent works [61, 67, 60, 23, 18, 26, 51] showed that multiple LLM agents or human-agent teams can improve upon single LLM in solving pure text-based tasks, such as creative writing, reasoning, and code generation. Other works [29, 17, 71] further explored agent selection or role assignment to improve the performance.

LLMs have also been applied to multi-agent cooperation for embodied tasks [1, 32, 36, 5]. Besides, Zhang et al. [64] proposed an intention inference framework to enhance the cooperation of LLM agents without explicit communication. Li et al. [24] investigated LLM-agents collaboration for Theory of Mind inferences tasks with a broadcast-only communication protocol and homogeneous policies. Zhang et al. [65] studied embodied multi-agent cooperation in the two-agent and the one-human-one-agent settings. Chen et al. [6] explored different fixed communication structures for multi-LLM-robots. Zhao et al. [70] and Chen et al. [4] organized the agents by predefined and fixed communication with a virtual manager. These initial explorations are limited to fixed team structures and are not optimized for communication efficiency. In contrast, our work explores the impact of deploying and optimizing organizational structures, allowing ≥3 agents in a team, for efficient multi-agent communication and cooperation.

Language models are sensitive to prompts. The format of the prompt can have a substantial influence on performance [11, 59, 72, 46, 75, 41]. Various research efforts have aimed at prompt optimization.

Method. We adopt the embodied LLM-agent architecture proposed by Zhang et al. [65] and expand it to enable organized teams of ≥3 agents to communicate, plan, and act in physical/simulated environments. Figure 2 illustrates our architecture. Borrowing insights from Zhang et al. [65], we adopt four standard modules: Configurator, Perception Module, Memory Module, and Execution Module. They are responsible for configuring the agents, translating environmental observations into text, storing & retrieving historical information, and executing actions, respectively (Fig. 2(a)).

Previous works focused on two-agent cooperation, in which case the communication can be simply treated as an extra action [32, 65]. In contrast, we aim to enable three or more agents to work in a team and cooperate through emergent organized communication. Thus we design the architecture with several features that facilitate organized multi-agent communication (Figure 2(b)):

• We disentangle the communication decision-making from the action decision-making by adopting two separate LLMs as Actor and Communicator.

• We impose an organizational structure for the agent team via prompting, i.e., including a textual description as part of the prompts for both the Actor and Communicator.

• LLM agents keep alternating between two phases during their task: the communication phase and the action phase. The standalone communication phase supports richer team structures and flexible communication patterns.

• During communication, agents take turns to communicate. An agent can choose to broadcast a message, select one recipient for a message, choose multiple recipients and send them distinct messages, or remain silent. Agents keep their own history of communication and can respond to messages from previous communications.

We leverage powerful LLMs to optimize the organizational prompt, borrowing insights from [62]. To do so, we introduce a dual-LLM framework to allow the multi-LLM-agent system to ponder and improve the organizational structure. Figure 3 illustrates the architecture of our framework. It consists of two LLMs:

• LLM critic: Inspired by the Actor-Critic method of reinforcement learning [21], we introduce an LLM critic to evaluate the team’s performance based on verbal feedback. The team critic takes as input the dialogue and action history of one episode. Then, the critic analyzes the input and reasons to extract and summarize the key steps that are believed to influence the performance. Also, the critic provides a textual evaluation of agents’ behaviors and the ranking of their leadership. See the prompts in Appendix A and technical details (including the ranking criteria) in Appendix B.1. • LLM coordinator: The LLM coordinator takes as input the outputs of the LLM critic as well as cost metrics (time to task completion and communication cost) of previous episodes from the environment. It reflects on these data and generates thoughts based on the analysis of the past episodes and the initial examples. With the reflection of organizational prompts and their performance, the coordinator proposes a new and different organizational prompt for the next episode. Please refer to Appendix A for the prompts and Appendix B.2 for the details of reflection.

For each new organizational prompt, we run for one episode and then return the dialogue and action history to the critic. By criticizing and reflecting on the prompts iteratively, the framework discovers more effective, novel organizational structures with self-improvement.

Discussion. Figure 7 reports the results and illustrates the behavior patterns for different LLM agents. The results support several observations. Even in a disorganized team, LLM agents love to tell others what to do. Leadership & assistance accounts for around > 50% of all the behavior (Figure 7(a)). However, other than telling others what to do, agents in the disorganized team do not show much cooperative behavior, for example, they would request for guidance in < 10% of the dialogues.

In contrast, when the team has a hierarchical organization, the lead LLM agent would presume a dominant role and give orders to others (amount to > 60% of their communication), while other members tend to follow and give fewer orders compared with the disorganized case. (Figure 7(b, c)). In such a team, agents tend to share and ask for more information, especially for the follower agents in the team. But still, the agents may fail to cooperate well, such as being lazy and confused about numbers, please see examples in Appendix F.6. and reduce unnecessary communication steps." This comparison highlights the role of the Critic and the importance of having a dual Criticize-Reflect architecture. For more results/prompts generated by the reflection process, please refer to Appendix H.

In addition, it is worth mentioning that LLMs are able to generate highly complex prompts that imply novel organizational structures that are rarely seen in human societies. We illustrate the communication patterns as team structures in Figure 9 together with the three novel structures proposed by Criticize-Reflect: (c) chain, (d) dual-leader, and (e) dynamic structures, which are the best structures of the three settings in Figure 8(b, c) respectively.

Conclusion. We develop a novel multi-LLM-agent architecture to facilitate communication and organize the embodied agent teams for enhanced cooperation. Moreover, we propose the Criticize-Reflect framework based on LLMs to generate more efficient organizational prompts. Extensive experiments with various group settings and organizational structures demonstrate that a hierarchically-organized team with a designated/elected leader has superior team efficiency, which can be further improved by Criticize-Reflect.

Limitations. The current work is performed in a single environment and lacks human evaluation. Future work shall extend to a broader set of environments, allowing human evaluation. As VirtualHome cannot hold hundreds of agents, future work can also explore larger organizations in other environments.