Can an AI learn when to speak and what to say as one skill, instead of bolting a timing rule onto a chat engine?
Can dialogue managers learn turn-taking and timing as joint behaviors?
This explores whether a dialogue system can learn *when* to speak (timing, interruptions, pauses) together with *what* to do in a turn as one learned skill, rather than having separate modules bolted together.
This explores whether a dialogue system can learn *when* to speak and *what* to do with its turn as one skill, instead of handing timing to a separate rule-based module. The corpus says yes, it can be built that way. It also shows that building it that way doesn't automatically make the timing good. The most direct example is Wan-Streamer, which merges language, audio and video into a single stream of tokens so that one Transformer learns response timing and turn management together. Turn-taking emerges from training rather than being engineered, with sub-second latency Can a single model learn when to speak and respond?.
The surprise comes from users. When people actually talked to Moshi, another unified full-duplex model, it interrupted aggressively and unpredictably. It also rarely produced the small signals humans rely on, like "mm-hm" backchannels or cues that it was still holding the floor. It didn't beat an old-fashioned pipeline that uses voice-activity detection to decide when the user has stopped talking Do spoken dialogue systems handle turn-taking as well as humans?. So a joint architecture makes joint learning possible, but the model only learns good timing if its training data and objectives reward it. A survey of group-chat and meeting agents fits this pattern: turn-taking and addressee prediction score above 80%, well ahead of emotion recognition. The authors note that part of that gap comes from better datasets, not easier problems Why do conversation agents handle turn-taking better than emotion recognition?.
The same argument for doing it jointly appears in a field that has nothing to do with voice. Conversational recommender systems have to decide whether to ask a question, recommend something now, or wait. Training those decisions as one policy beats splitting them up, because separate components can't pass learning signals to each other or plan the conversation as a whole Can unified policy learning improve conversational recommender systems?. Timing matters in text chat too. In one study, agents that adjusted their pacing to the context made people feel more heard and led them to share more about themselves Can pacing strategies make conversational agents feel more human?.
The question also leads somewhere bigger. Timing isn't only about milliseconds. It's also about knowing when to step in. Proactive dialogue, where the system offers relevant information before being asked, can cut conversations by up to 60%, yet it barely appears in AI datasets Could proactive dialogue make conversations dramatically more efficient?. LLM agents are reactive by design and can't take the initiative Why can't conversational AI agents take the initiative?. Preference training such as RLHF makes this worse: it cuts clarifying questions and understanding checks to 77.5% below human levels Does preference optimization harm conversational understanding?. And the human habits that keep a conversation running smoothly, like fixing misunderstandings or handing off a topic, are social actions that next-word prediction doesn't reward Why don't language models develop conversation maintenance skills?.
The takeaway: the architecture problem is mostly solved, and the open problem is the training signal. A model can learn timing and content together, but it will only learn *good* timing if it's rewarded for social moves like backchanneling, holding the floor, stepping in at the right moment and checking that it was understood. Today's datasets and alignment methods rarely reward any of these.
Sources 9 notes
Wan-Streamer represents language, audio, and video as one interleaved causal token stream, allowing response timing and turn management to be learned jointly within a single Transformer rather than engineered as separate modules, achieving sub-second latency.
A user study comparing Moshi (full-duplex E2E) and a VAD-based cascaded system found both rarely backchannel or signal floor-holding cues, while Moshi over-interrupts unpredictably. Even unified causal architectures did not produce better turn management than rule-based cascades.
Across 70+ papers on group-chat and meeting agents, turn-taking and addressee-selection tasks exceed 80% accuracy while emotion, personality and engagement recognition average 60%. The gap partly reflects dataset quality differences rather than pure capability limits.
Research shows that formulating attribute-asking, item-recommending, and timing decisions as a single graph-based RL policy achieves better joint optimization than isolated components. Separation prevents gradient signals from informing one another and fails to optimize conversation trajectory holistically.
A 50-person study found that conversational agents using dynamic pacing strategies outperformed static-pacing baselines on perceived listening quality, human-likeness, and user self-disclosure in text-based career and relationship support scenarios.
Show all 9 sources
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Research shows LLMs including ChatGPT cannot initiate topics, plan strategically, or lead conversations because their training optimizes for responding to queries, not creating dialogue from agent goals. This passivity is reinforced by alignment objectives and masked by fluent-sounding outputs.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
Humans keep conversations smooth through implicit techniques like reference repair and topic hand-off that sustain relational interaction, not convey information. Language models don't develop these because training signals reward information prediction, not relational work.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Proactive Conversational Agents with Inner Thoughts
- Proactive Conversational Agents in the Post-ChatGPT World
- DiscussLLM: Teaching Large Language Models When to Speak
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
- Rethinking Conversational Agents in the Era of LLMs: Proactivity, Non-collaborativity, and Beyond
- Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents
- When Single-User-Oriented LLM-based Assistants Involve Others: A Scoping Review of Pathways, Risks, and Responses