Would an AI meeting assistant read the room better if it could hear tone of voice and see faces, not just read the transcript?
Do multi-modal datasets improve emotion recognition in group conversation agents?
This explores whether training AI agents on richer data, such as voice, video and facial cues alongside text, helps them read emotions in group settings like meetings and group chats, where they currently struggle.
This explores whether giving group conversation agents richer data, such as tone of voice, facial expressions and video alongside the transcript, helps them pick up on how people feel. The short answer is that the collection doesn't contain a direct test of multimodal versus text-only emotion recognition in groups. It does show that emotion recognition is where these agents are weakest, and it suggests that the data is a big part of why.
The clearest evidence comes from a survey of more than 70 papers on meeting and group-chat agents Why do conversation agents handle turn-taking better than emotion recognition?. Agents do well at the mechanics of conversation: predicting who speaks next and working out who is being addressed both exceed 80% accuracy. Recognizing emotion, personality and engagement averages around 60%. The survey traces part of that gap to the datasets rather than to model ability alone. Turn-taking has clean, objective labels. Emotion labels in group settings are sparse, noisy and often disputed. That shifts the question: more types of data may help less than better-labeled data.
Work on alignment suggests why non-text signals still matter. A systematic review finds that matching someone's words drives task efficiency, while emotional and prosodic alignment (rhythm and tone of voice) drives warmth and trust Do different types of alignment serve different conversational goals?. If people express and read feelings partly through the voice, a text-only agent misses part of the signal. Even in pure text, timing carries emotional weight: agents that adjust their response pacing to the context were rated as better listeners, and users told them more about themselves Can pacing strategies make conversational agents feel more human?. Treating emotion as a trajectory that unfolds over the conversation, rather than a label on a single message, also reveals patterns that snapshot analysis misses Can tracking dialogue dimensions simultaneously reveal hidden conversation patterns?.
Recognizing an emotion is also not the same as responding to it well. LLM therapists tend to jump to problem-solving when users share feelings, a habit likely reinforced by RLHF's push toward helpfulness Do LLM therapists respond to emotions like low-quality human therapists?. One fix uses a simulated user's emotional trajectory as the reward signal during training, which made models noticeably more empathic without hurting dialogue quality Can emotion rewards make language models genuinely empathic?. A deeper argument holds that the relational work of conversation, such as smoothing over misunderstandings and handing off topics, is social action rather than information. Training that only rewards predicting the next piece of information won't produce it Why don't language models develop conversation maintenance skills?.
The takeaway: multimodal data is a plausible lever, but the corpus points to label quality, emotion tracked over time, and training signals that reward relational behavior as at least as important. An agent that sees every facial expression but was trained only to be helpful may still answer someone's frustration with a bulleted action plan.
Sources 7 notes
Across 70+ papers on group-chat and meeting agents, turn-taking and addressee-selection tasks exceed 80% accuracy while emotion, personality and engagement recognition average 60%. The gap partly reflects dataset quality differences rather than pure capability limits.
A 2020–2025 systematic review shows lexical alignment drives task efficiency and comprehension, while emotional and prosodic alignment drive relational warmth and trust. Conflating them in design produces category errors—cold customer-service bots and evasive mental-health assistants.
A 50-person study found that conversational agents using dynamic pacing strategies outperformed static-pacing baselines on perceived listening quality, human-likeness, and user self-disclosure in text-based career and relationship support scenarios.
Conversational DNA encodes four simultaneous dimensions—linguistic complexity, emotional trajectories, topic coherence, and conversational relevance—as temporal streams. The reverse Turing test finding showed expert assessments of AI diverged sharply, suggesting conversational structure shapes interpretation as much as content.
Using the BOLT framework, researchers found LLMs offer solution-focused advice during emotional disclosure—a hallmark of low-quality therapy—yet also reflect more on client needs and strengths than typical poor human therapy, creating an unusual hybrid profile likely driven by RLHF's helpfulness bias.
Show all 7 sources
RLVER uses a simulated user's emotion trajectory as an RL reward signal, enabling GRPO to deliver stable empathy improvements while maintaining dialogue quality—countering the typical trade-off between preference optimization and conversational grounding.
Humans keep conversations smooth through implicit techniques like reference repair and topic hand-off that sustain relational interaction, not convey information. Language models don't develop these because training signals reward information prediction, not relational work.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- ChatGPT Reads Your Tone and Responds Accordingly -- Until It Does Not -- Emotional Framing Induces Bias in LLM Outputs
- RLVER: Reinforcement Learning with Verifiable Emotion Rewards for Empathetic Agents
- Conversational Alignment with Artificial Intelligence in Context
- Multi-Party Conversational Agents: A Survey
- Can LLMs identify and repair ruptures? Comparison between clinician practices and LLM behaviors
- Proactive Conversational Agents with Inner Thoughts
- H2HTalk: Evaluating Large Language Models as Emotional Companion
- The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs