Why do conversation agents handle turn-taking better than emotion recognition?

A survey of multi-party conversational agents (MPCAs) — systems built to hold dialogue with more than two participants, as in group chats, team meetings, online gaming and customer-support forums — organizes the field into a three-part taxonomy: State of Mind Modeling (emotion recognition, personality recognition, engagement detection and dialog-act/intention recognition), Semantic Understanding (dialogue summarization, conversation disentanglement, discourse analysis), and Agent Action Modeling (turn detection, addressee selection, response selection). Reviewing benchmark results across these three themes, the authors report a sharp split: "Tasks involving agent action modeling (e.g., turn detection, addressee and response selection) often exceed 80% accuracy, while most other tasks average around 60%." Some models reach even higher on narrow benchmarks — a dense-retrieval model attains 91% recall on the Ubuntu IRC corpus — but the authors note such results are "confined to niche domains, limiting their real-world applicability and transferability."

The taxonomy follows from a view of socially intelligent dialogue borrowed from Social Identity Theory (Tajfel and Turner, 1979): an agent's actions — when to speak, whom to address, what to say — are shaped by its own and others' mental states, and those actions in turn shape mental states. Semantic understanding mediates between the two, letting an agent interpret the social cues and context that mental-state inference and action planning both depend on. The survey frames the accuracy gap as partly a benchmarking artifact rather than purely a capability gap: action-modeling datasets (who speaks next, who is addressed) are more structurally tractable and better-resourced than the datasets for emotion, personality and engagement, and multi-modal datasets combining facial expression, gaze and vocal tone — which the authors call essential for capturing mental states — remain scarce.

The nearest notes in this vault address two-party, single-user dialogue. Can conversation structure predict dialogue success better than content? finds that structural features of a conversation's trajectory predict satisfaction about as well as its content, and Why can't conversational AI agents take the initiative? identifies proactivity as a missing capability across single-agent LLMs. This survey extends that passivity question to the group setting: agent action modeling (turn detection, addressee selection) is precisely the mechanism by which an agent would take initiative in a multi-party conversation — deciding when to speak and to whom — and the survey's finding that this is the best-performing of the three task families suggests the mechanics of taking a turn are more tractable than knowing what the turn should accomplish socially.

The survey does not evaluate any single integrated MPCA system end-to-end; the 80%-versus-60% comparison is drawn across separate benchmarks measuring separate skills in isolation, not one agent exercising all three capabilities together in a real multi-party conversation. It also does not establish that action-modeling accuracy transfers outside the narrow domains (meeting corpora, IRC logs) where it was measured. The authors' further recommendation — that integrating the three themes, and especially adding Theory of Mind, would improve MPCA performance — is stated as their own intuition rather than as a tested result.