Do spoken dialogue systems handle turn-taking as well as humans?

"Talking Turns" reports what it calls the first empirical user study evaluating audio foundation models' ability to perform turn-taking during live spoken conversation, rather than only transcribe or generate text. The study tests the "full-duplex E2E spoken dialogue system Moshi and VAD-based cascaded dialogue system" and reports, in the authors' own summary: "(a) Both dialogue systems sometimes do not speak up even when user wants to yield its turn. (b) Moshi interrupts too aggressively and its interruptions occur at unlikely instances. (c) Both dialogue systems rarely backchannel. (d) Both dialogue systems do not give users enough cues about when they want to keep the conversation floor."

Prior evaluation relied on "corpus level statistics" comparing how well the aggregate distribution of turn-taking events in generated dialogue matches ground truth, but the authors write these "fail to evaluate the exact timing when a turn-taking event happens within the local context." To fix this, the paper trains a supervised "judge" model on human-human conversation data (drawn from Switchboard) that predicts, every 40ms, what turn-taking event — turn change, backchannel, interruption — "an ideal system should make," then uses that judge's output as an automated, timing-centric stand-in for human relevance judgments, since collecting live human ratings for every event is "expensive and time-consuming." The underlying vocabulary it measures against includes inter-pausal units, silence, pause (within-speaker silence), gap (between-speaker silence), and overlap.

The test subject Moshi is the same full-duplex, single-stream design that Can a single model learn when to speak and respond? holds up as the architecture that should make turn management a learned, emergent behavior rather than an engineered VAD threshold. This study is a direct behavioral check on that claim, and the result is mixed: a unified causal stream does not yet produce better turn-taking than the VAD-based cascade it is compared against — both score poorly on backchanneling and floor-holding cues, and Moshi specifically over-interrupts. The finding also complicates Why do conversation agents handle turn-taking better than emotion recognition?: that survey's high numbers come from offline classification benchmarks (who speaks next, who is addressed), not from a system actually producing turns live, and the backchanneling and silence-handling failures here echo the behaviors catalogued in What pacing moves make conversational agents sound like active listeners? — precisely the pacing moves both tested systems rarely perform.

The judge model's own agreement with human turn-taking judgments "is below 80% for many metrics," so its scores are an imperfect proxy, not a ground-truth measure of conversational quality. The user study "tests only a few audio FMs," and the authors note participants "were primarily the authors and their research colleagues," raising the possibility of evaluator bias; backchannel detection also relies on heuristics that "may miss certain backchannels." The excerpt does not establish that these specific failure patterns generalize beyond Moshi and this one cascaded baseline, but it does establish that turn-taking competence cannot be assumed from a model's having full-duplex or real-time architecture alone — it has to be measured directly against live interaction.