Multi-Party Conversational Agents: A Survey

Multi-party Conversational Agents (MPCAs) are systems designed to engage in dialogue with more than two participants simultaneously. Unlike traditional two-party agents, designing MPCAs faces additional challenges due to the need to interpret both utterance semantics and social dynamics. This survey explores recent progress in MPCAs by addressing three key questions: 1) Can agents model each participants’ mental states? (State of Mind Modeling); 2) Can they properly understand the dialogue content? (Semantic Understanding); and 3) Can they reason about and predict future conversation flow? (Agent Action Modeling). We review methods ranging from classical machine learning to Large Language Models (LLMs) and multi-modal systems. Our analysis underscores Theory of Mind (ToM) as essential for building intelligent MPCAs and highlights multi-modal understanding as a promising yet underexplored direction. Finally, this survey offers guidance to future researchers on developing more capable MPCAs.

Introduction. Developing agents for conversational understanding and natural dialogue generation has been a longstanding research agenda within the NLP community (Wang et al., 2024a). Traditionally, most conversational agents have been designed for two-party interactions (Zheng et al., 2022). However, realworld conversations—such as group chats, team meetings, online gaming, and customer support forums—often involve more than two participants. As a result, there has been growing interest in developing agents capable of engaging with multiple participants simultaneously, commonly referred to as Multi-Party Conversational Agents (MPCAs). Compared to traditional two-party systems, building MPCAs presents greater challenges due to the complex dynamics in multi-party interactions. Effective MPCAs must simultaneously infer participants’ mental states (Cohen et al., 2024), comprehend user content (Ma et al., 2023a), and anticipate future dialogue flow (Houde et al., 2025). Mastering these capabilities is crucial for enabling collaborative tools, socially adept robots, and intelligent assistants. For instance, Figure 1 illustrates a two-speaker interaction with an agent, highlighting core challenges such as emotional variance, evolving latent states, and context-sensitive response planning (Wei et al., 2023; Houde et al., 2025). While recent advances in conversational AI (AI, 2024; OpenAI, 2024; Qwen et al., 2025; Jiang et al., 2024a) have improved these components individually, integrating them into fully functional MPCAs still remains an open challenge (Zhao et al., 2023; Tan et al., 2023). This research gap has recently attracted significant attention from the community, leading to a surge in efforts to design multi-party conversational systems (Ganesh et al., 2023). In synergy with with recent advances in generative AI, particularly Large Language Models (LLMs), most of these efforts have focused on leveraging LLMs for this domain (Wang et al., 2024c; Tan et al., 2023). As such, this survey aims to serve as a comprehensive resource for AI researchers interested in developing MPCAs. We review over 70 research articles relevant to multi-party conversations and categorize their contributions into three broad themes:

Method. 2 A Thematic Taxonomy To organize this survey paper, we create a taxonomy of existing research efforts on multi-party conversations (MPC) by categorizing them into three core themes: State of Mind Modeling, Semantic Understanding, and Agent Action Modeling. This taxonomy reflects essential capabilities for humanlike social communication in group settings. Indeed, a socially intelligent agent must determine when to speak, whom to address, and what to say, collectively referred to as actions. According to Social Identity Theory (Tajfel and Turner, 1979), such actions are influenced by the agent’s own and others’ mental states, while reciprocal effects also exist, where actions can shape mental states (Hatfield et al., 1993). Semantic understanding (Kopp and Krämer, 2021) further mediates these dynamics by enabling agents to interpret social cues and context (Rijpma et al., 2023). As illustrated in Figure 1, these three components interact dynamically throughout the course of a conversation. Next, we formalize our proposed taxonomy by mapping existing MPC tasks to these three themes as follows: tasks aligned with State of Mind Modeling include emotion recognition, personality detection, dialog act recognition, and engagement detection; tasks supporting Semantic Understanding include dialog summarization, conversation disentanglement, discourse analysis, and representation Understanding the state of mind of participants is essential for developing socially intelligent multiparty conversational agents. But what exactly constitutes a participant’s state of mind? Our survey identifies four key components that together provide a comprehensive understanding of participants’ mental states during conversation: emotion recognition, engagement detection, personality recognition, and conversational intentions (i.e., dialog act recognition). These essential components of mental states are defined and modeled by legacy psychological frameworks, such as the OCEAN model for personality traits (Kasap et al., 2009) and the OCC model (Ortony et al., 1988) for emotions (Irfan et al., 2020; Gebhard, 2005). Below, we discuss these four key components pivotal to State of Mind Modeling.

Emotion recognition (ER) is the task of classifying emotions as positive (e.g., joy) or negative (e.g., anger) (Ortony et al., 1988). It helps MP- CAs to perceive and respond empathetically. Initial approaches did contextual modeling (Sun et al., Engagement detection measures the interaction level of participants in conversations, essential for managing interactions and turns between participants. Despite being a difficult problem, recent multi-modal methods used visual and audio cues, enhancing performance by continuously tracking participants’ behaviors (Bohus and Horvitz, 2009; Xu et al., 2013). Other methods increase accuracy by integrating group interaction and spontaneous conversation (Salam and Chetouani, 2015; Fedotov et al., 2018). Recently, multi-modal LLMs have shown promise, offering interpretability and flexibility for engagement detection (Ma et al., 2024). However, effectively using multi-modal frameworks and LLMs for this task remains a challenge.

Personality recognition (PR) helps MPCAs identify personality traits of participants from conversation modalities (video, text, etc.), commonly using the Big-Five personality model: extraversion, agreeableness, conscientiousness, neuroticism, and openness (Cunningham, 1977). Initial text-based methods analyzed and aggregated predictions over dialogues to predict personality (Ríssola et al., 2019). Later, transformers like BERT and RoBERTa enhanced PR through attention mechanisms (Jiang et al., 2019). Other approaches framed PR as a natural language inference task using emotion annotations (Wen et al., 2024). Multi-modal methods for PR integrate vision, audio, and behavioral cues to solve the task. Initial methods clustered co-occurring non-verbal behaviors (Okada et al., 2015), while advanced techniques employ convolutional neural networks (CNNs) (Kimura and Okada, 2020) and attentionbased models (Lin and Lee, 2018).

Discussion. So far, we have discussed a range of tasks within our thematic taxonomy, each associated with multiple benchmark datasets for evaluation. Table 1 summarizes the most widely used benchmarks along with state-of-the-art (SOTA) model performance. Further details are provided in Appendixes B and C. These datasets vary in size, domain, and modality and are typically designed to evaluate a single MPCA skill, such as emotion recognition (Poria et al., 2019), personality detection (Sanchez-Cortes et al., 2012), or dialogue summarization (Zhong et al., 2021), which may limit their generalizability. Domain specificity also presents a challenge. For example, the AMI Meeting Corpus (Carletta et al., 2005), despite offering annotations for multiple tasks, is underutilized due to its narrow focus on formal product design meetings (see Table 3). A notable limitation is the scarcity of robust multi-modal datasets that integrate non-verbal cues such as facial expressions, gaze, or vocal tone—essential for capturing participants’ mental states and enriching semantic context. Analysis of SOTA models across benchmarks reveals significant performance disparities. Tasks involving agent action modeling (e.g., turn detection, addressee and response selection) often exceed 80% accuracy, while most other tasks average around 60%. Although certain models achieve near-perfect scores on specific benchmarks (e.g., Dense Retrieval models attaining 91% recall on Ubuntu IRC (Lan et al., 2022; Kummerfeld et al., 2019)), such datasets are often confined to niche domains, limiting their real-world applicability and transferability. Moreover, SOTA models for multimodal benchmarks remain scarce, despite the critical role of cross-modal information in enhancing MPCA performance. Overall, these observations underscore the limitations of current benchmarking practices and highlight the need for more comprehensive, diverse, and multi-modal datasets to advance future research.

Building intelligent MPCAs still remains an open challenge. While our survey highlights significant progress, there remain large gaps in performance. We recommend focusing on three major research thrusts as follows to tackle these challenges.

Theory of Mind: Theory of Mind (ToM) refers to the ability to understand, reason, and build beliefs about what participants might be thinking or feeling at each step of a conversation, even if those thoughts and feelings are not explicitly expressed (Premack and Woodruff, 1978). ToM enables agents to reason and infer the beliefs and intentions of their own and other participants. ToM is an essential skill for MPCAs as it falls under the umbrella of State of Mind Modeling (Vanderlyn et al., 2025). Techniques like Mind- Dial (Qiu et al., 2024) and benchmarks like MuMA- TOM (Shi et al., 2025) and KokoMind (Shi et al., 2023) show promise in modeling of ToM in conversations. However, the benchmarking is done on synthetic datasets, thereby not emulating realistic multi-party settings for ToM. There are works probing LLMs for ToM reasoning capabilities (Chen et al., 2024; Hou et al., 2024), but studies show that scaling up LLMs does not always improve ToM reasoning (Sclar et al., 2023). On standard ToM tasks, some LLMs achieve near-human performance (Ma et al., 2023b), suggesting the ability to track beliefs and intentions. Nevertheless, there is an ongoing debate as to whether these models truly reason or simply memorize patterns about mental states (Ullman, 2023; Kosinski, 2024). Latest benchmarks, ToMATO (Shinoda et al., 2025), reveal gaps for strong models like GPT-4. Also, there is little work in integrating ToM capabilities into other MPCA tasks. Our intuition is that such integration will improve MPCAs performance.

Conclusion. The development of multi-party conversation agents has come a long way from text-based machine learning to multi-modal and LLM-based approaches. In this survey paper, we discussed the recent literature on MPCAs by defining a thematic taxonomy and categorizing works/tasks according to it. Our survey provides details on how different methods solve each problem of MPCAs. It also provides insights into the limitations and challenges faced when developing MPCAs. Finally, we discuss future directions for this field of research, specifically how Theory of Mind, multi-modal fusion, and grounding, and better benchmarks can help us develop better MPCAs. Addressing these aspects will advance MPCAs toward genuinely engaging and socially aware interactions, bridging the gap between current systems and human-like conversational intelligence.

Limitations. Our survey primarily focuses on research published after 2021, which may exclude some foundational or emerging contributions outside this period. Given the extensive body of related work, we acknowledge the possibility of overlooking equally significant studies. The future directions we propose are based on our interpretation of current research trends and supported by cursory experiments, which may not capture all perspectives. Additionally, many works we reviewed rely on custom datasets and models, some of which do not align with recent standard benchmarks. This reliance may introduce biases in our analysis. We recommend further, more comprehensive research to cross-verify our findings.