Chatbots pass standard conversation tests but still can't say a well-timed 'mm-hm' while you're talking — why not?
Why do dialogue benchmarks show 80% accuracy but systems fail at backchanneling?
This explores why spoken dialogue systems can score well on standard benchmarks yet still fail at backchanneling, the small 'mm-hm', 'right' and 'yeah' signals a listener gives while someone else is talking. The corpus doesn't contain the specific 80% figure, but it does explain why that kind of gap opens up.
This explores why dialogue systems that look strong on benchmarks still can't do the small listener signals ('mm-hm', 'right', a well-timed 'yeah') that keep human conversation flowing. The corpus has no source for the exact 80% number. It does point to a clear pattern: dialogue benchmarks mostly score what a system says when it's its turn. Backchanneling is about what a system does while it's someone else's turn, so the benchmarks mostly don't measure it.
The most direct evidence comes from a user study of real spoken systems Do spoken dialogue systems handle turn-taking as well as humans?. Both a modern full-duplex model (Moshi, which listens and speaks at the same time) and an older rule-based pipeline almost never backchanneled or signaled 'I'm still talking, hold on.' Moshi also interrupted at unpredictable moments. The surprising part is that the more advanced design, which in principle could produce a timely 'mm-hm', managed turn-taking no better than the old pipeline. The ability to listen while speaking doesn't come with conversational timing included. A related reason this goes unnoticed: speech benchmarks are still mostly scored on transcripts How can dialogue systems handle the noise in speech input?. Once audio is turned into text, the timing, overlap and tone that carry a backchannel are gone, so a benchmark can't reward or penalize them.
This isn't only a speech problem. The same blind spot shows up in text dialogue. Models trained on rewards for the next reply learn to give one helpful-looking answer instead of taking part in the conversation as it unfolds Why do language models respond passively instead of asking clarifying questions?. Accuracy measured one message at a time can also be misleading. Models that score around 90% on single-message tasks fall to about 65% when the same information arrives over several turns Why do AI assistants get worse at longer conversations? Why do language models fail in gradually revealed conversations?. Two other everyday human habits are also nearly absent from AI systems and benchmarks: adopting the other person's word choices Why don't conversational AI systems mirror their users' word choices? and offering useful information before being asked Could proactive dialogue make conversations dramatically more efficient?. Backchanneling belongs with these. They are all ways a speaker coordinates with the other person, and none of them shows up in a score on single answers.
Two findings suggest the fix is harder than adding more 'mm-hm's. First, a system has to backchannel while the speaker is still mid-sentence, based on partial and noisy input. Speech recognition in real settings gets 15–30% of words wrong, so a system that commits to one reading of what it heard will misfire. Systems need to keep track of several possible interpretations at once Why do dialogue systems need probabilistic reasoning?. Second, more backchannel isn't always better. In a CHI 2025 study, frequent backchannels from an LLM voice agent lowered older adults' ratings, because age-related slowing in how the brain processes sound made the rapid cues feel like interruptions Does frequent backchannel from voice agents help or hurt older adults?.
The takeaway: good backchanneling isn't a fixed behavior a benchmark could check off. It depends on the listener, who might need more cues, fewer, or slower ones. A benchmark scored on transcripts with one fixed answer per item can't measure that. So the gap between benchmark scores and real conversation is probably less about weak models and more about the benchmarks measuring a different skill.
Sources 9 notes
A user study comparing Moshi (full-duplex E2E) and a VAD-based cascaded system found both rarely backchannel or signal floor-holding cues, while Moshi over-interrupts unpredictably. Even unified causal architectures did not produce better turn management than rule-based cascades.
Direct speech-to-speech models achieve 226 ms latency and infer articulatory physics without phonetic labels, enabling cross-lingual transfer. Yet evaluation benchmarks remain anchored to transcription, leaving comprehension and reasoning capabilities unmeasured.
CollabLLM demonstrates that standard RLHF training optimizes for immediate helpfulness, discouraging models from asking clarifying questions or offering multi-turn insights. Multi-turn-aware rewards that estimate long-term interaction value enable active intent discovery and genuine collaboration.
LLMs perform at 90% accuracy with single-message instructions but drop to 65% across natural conversation. Models lock into early guesses when information arrives gradually and cannot course-correct, a behavior induced by RLHF training that rewards helpfulness over clarification.
Across 200,000+ conversations, all major LLMs show 39% average performance drop in multi-turn settings due to locking into incorrect early guesses. Agent mitigations recover only 15-20% of this loss.
Show all 9 sources
Response generation models fail to adapt vocabulary toward users' lexical choices, a phenomenon central to human rapport and clarity. Post-training via DPO on coreference-identified preferences can teach models in-context convention formation.
Simulations show proactivity—providing relevant information without being asked—cuts dialogue turns by 60% in medium-complexity domains. This behavior mirrors human conversation and Grice's maxims but is almost entirely absent from AI datasets and research benchmarks.
Real-world speech recognition achieves 15-30 percent error rates in noisy environments, making deterministic flowchart dialogue systems unworkable. POMDP-based systems handle this by maintaining belief distributions over user intent rather than committing to single interpretations.
A CHI 2025 study found that frequent backchannel cues from an LLM voice agent reduced older adults' experience ratings. Age-related slowing in central auditory processing made rapid backchannel signals harder to distinguish and integrate into conversation, feeling disruptive rather than engaging.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Proactive Conversational Agents with Inner Thoughts
- Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics
- Are LLMs All You Need for Task-Oriented Dialogue?
- LLMs Get Lost In Multi-Turn Conversation
- DiscussLLM: Teaching Large Language Models When to Speak
- Proactive Conversational Agents in the Post-ChatGPT World
- Toward Enabling Natural Conversation with Older Adults via the Design of LLM-Powered Voice Agents that Support Interruptions and Backchannels