Can we trust an AI 'judge' to score when a voice assistant should speak, pause, or interrupt — or does that still need human ears?

This explores whether trained models that score when a spoken AI should speak, pause or interrupt can be trusted, or whether turn-taking timing still needs human judgment to evaluate. The short answer is that the corpus has no paper that directly validates a supervised judge for turn-taking timing. What it does have points to a gap: the evidence that tells us most about timing quality comes from people, not from automated scorers.

The clearest example is a user study comparing Moshi, a full-duplex model that listens and speaks at the same time, with an older cascaded system that uses voice-activity detection to decide when to talk Do spoken dialogue systems handle turn-taking as well as humans?. People noticed things a timing metric could easily miss. Moshi interrupted unpredictably, and neither system produced the small signals humans use to manage a conversation, such as backchannels like "mm-hm" or cues that say "I'm not done yet." Meanwhile, unified streaming models such as Wan-Streamer report sub-second latency and treat turn-taking as something the model learns on its own Can a single model learn when to speak and respond?. That shows the risk for any judge. Speed is easy to measure, but a fast response can still feel rude. A judge trained to reward low latency would grade the wrong thing.

Turn-taking also looks easier than it is. Across 70+ papers on group-chat and meeting agents, turn-taking and addressee-selection tasks score above 80% accuracy, while emotion and engagement recognition sit around 60% Why do conversation agents handle turn-taking better than emotion recognition?. The survey notes that part of this gap comes from cleaner datasets rather than real capability. If a supervised judge is trained on those clean labels, its high agreement scores may tell you more about the benchmark than about how well the judge works. Timing also depends on context: in support chats, pacing that adapted to the conversation beat fixed pacing on perceived listening quality and on how much users opened up Can pacing strategies make conversational agents feel more human?. There is no single correct pause length for a judge to learn. The right gap depends on what was just said and who said it.

Two nearby lines of work show what a more reliable judge might need. Conversation-forecasting research found that small models trained to state their uncertainty, and to abstain when unsure, matched models ten times larger Can models learn to abstain when uncertain about predictions?. Predicting the moment a turn ends is a forecasting problem, so a judge that can say "this one is ambiguous" is probably worth more than one that always gives a confident score. Work on user simulators suggests a second safeguard: their realism was checked in three independent ways, with crowdworkers, discriminator models and distribution matching, rather than with one automated scorer Can controlled latent variables make LLM user simulators realistic?.

Here's what you might not expect. The behaviors that matter most for good turn-taking, like backchannels, clarifying questions and checks that the listener understood, are the ones current training tends to remove. Preference optimization cut these grounding moves to 77.5% below human levels Does preference optimization harm conversational understanding?. Proactive turns that skip unnecessary back-and-forth are almost absent from benchmarks Could proactive dialogue make conversations dramatically more efficient?. A judge trained on today's data could learn to reward the very conversation style that makes these systems feel off. For now, human user studies remain the most trustworthy check.