Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics

The recent wave of audio foundation models (FMs) could provide new capabilities for conversational modeling. However, there have been limited efforts to evaluate these audio FMs comprehensively on their ability to have natural and interactive conversations. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Inspired by this, we ask whether the recently proposed audio FMs can understand, predict, and perform turn-taking events? To answer this, we propose a novel evaluation protocol that can assess spoken dialog system’s turn-taking capabilities using a supervised model as a judge that has been trained to predict turn-taking events in human-human conversations. Using this protocol, we present the first comprehensive user study that evaluates existing spoken dialogue systems on their ability to perform turn-taking events and reveal many interesting insights, such as they sometimes do not understand when to speak up, can interrupt too aggressively and rarely backchannel. We further evaluate multiple open-source and proprietary audio FMs accessible through APIs on carefully curated test benchmarks from Switchboard to measure their ability to understand and predict turn-taking events and identify significant room for improvement. We will open source our evaluation platform to promote the development of advanced conversational AI systems.

Introduction. When humans interact with an audio foundation model (FM), it is a two-way communication. Similar to human-human conversations, the model listens and speaks, and more importantly, does both at the same time. Turn management is at the core of real-world conversations. This means that when the user speaks, the system should know when to listen and when to speak (Gravano & Hirschberg, 2011). The system should additionally provide subtle cues, referred to as backchannels (Fujie et al., 2005), to indicate that it is “listening” and make relevant follow-up interruptions (Lee & Narayanan, 2010), ensuring that the conversation feels interactive. Furthermore, when the system speaks, it should formulate its output in a way that conveys to the user whether it wants to keep the conversation floor (Ekstedt et al., 2023) and also appropriately address user interruptions (Ma et al., 2024).

Commercial voice assistants (Li et al., 2017) use traditional silence duration-based end-of-turn prediction models for turn management. However, this is insufficient for having natural conversations because silence is often not the main cue for humans to switch turns. In fact, research shows that silence within the same speaker’s turns (pauses) tends to be longer than silence between different speaker’s turns (gaps) (Ten Bosch et al., 2005). Hence, this continues to remain a challenging task for human-machine interaction. As a result, there has been a lot of interest in automatically predicting turn-taking events. Prior research works (Hara et al., 2018; Li et al., 2022; Fujie et al., 2005; Lee et al., 2008) have explored automatically predicting turn change and backchanneling from text, audio, and multimodal input. More recently, audio FMs (Xie & Wu, 2024; D ́efossez et al., 2024) have been proposed that can automatically take turns while interacting with the end user.

However there have been limited efforts to evaluate these FMs on their conversational capabilities. Furthermore, this requires more than simply testing on standard speech recognition (ASR), text-tospeech (TTS), and text-based dialogue benchmarks. To engage in meaningful conversation with the end user, we would want the FMs to additionally perform a fluent succession of turns without too much overlapping speech or long stretches of silence. Finally, the model should be able to engage in dynamic and fluid conversations where it should use cues to confirm listening and encourage the speaker, and it should listen for cues to continue or to be interrupted while speaking. To the best of our knowledge, there has not been any prior effort to empirically evaluate the turn-taking capabilities of audio FMs. As a result, the community still lacks a fine-grained understanding of the relative merits and limitations of different audio FMs on their conversational capabilities. Motivated by this, our work presents the first empirical investigation to evaluate audio FMs on their ability to understand, predict, and perform turn-taking events. Our key contributions are:

Contribution 1: An evaluation protocol to assess audio FM’s capability to perform turn-taking events: Prior work (Nguyen et al., 2023) uses corpus level statistics for automatic evaluation. While these metrics capture how well the distribution of the turn-taking events globally in the generated dialogue matches with ground truth, they fail to evaluate the exact timing when a turn-taking event happens within the local context. We propose evaluating the timing of turn-taking events by training a model on human-human conversations to effectively predict the turn change, backchannel, and interruption. We use this predictor model as the judge since getting human relevance judgments for each turn-taking event is expensive and time-consuming. Using the judge model, we propose automated metrics corresponding to each of the core conversational abilities, thus building a comprehensive suite of diverse metrics that can empower a systemic understanding of any audio FM with conversation capability on its ability to manage turns. Finally, we plan to publicly release our evaluation platform so that researchers can easily test their own pre-trained audio FMs.

Contribution 2: Interesting insights about existing spoken dialogue systems: We run a user study with different spoken dialogue systems, namely full-duplex E2E spoken dialogue system Moshi and VAD-based cascaded dialogue system and evaluate these systems using both corpus level statistics and our proposed metrics, unveiling many interesting observations: (a) Both dialogue systems sometimes do not speak up even when user wants to yield its turn. (b) Moshi interrupts too aggressively and its interruptions occur at unlikely instances. (c) Both dialogue systems rarely backchannel. (d) Both dialogue systems do not give users enough cues about when they want to keep the conversation floor.

Related work. 2 Related Study Turn-Taking Prediction: There have been many prior works on predicting turn change (Gravano & Hirschberg, 2012; Hara et al., 2018; Li et al., 2022), backchannel (Fujie et al., 2005) and interruptions (Lee & Narayanan, 2010; Lee et al., 2008) from audio, text and other multimodal information (Morency et al., 2010; Scherer et al., 2012) (detailed related work discussion in Appendix A.1). The Voice Activity Projection (VAP) models (Ekstedt & Skantze, 2022a; Inoue et al., 2024) are trained in an unsupervised manner on spoken dialogue data to predict future “speech” activity. Other recent works (Wang et al., 2024) use pseudo labels to predict turn-taking and backchanneling locations in spoken dialogue by integrating a neural acoustic model with a large language model (LLM). Recently, audio FMs like Moshi (D ́efossez et al., 2024) and GPT-4o 1 have been proposed that can perform turn-taking to achieve simultaneous real-time 2-channel conversation. However, these FMs have still not been quantitatively evaluated on their ability to perform turn-taking events.

Benchmarks for Audio FM: Benchmarks like Dynamic SUPERB (Huang et al., 2023) and AIR- Bench (Yang et al., 2024) have been valuable for evaluating audio FMs on standard speech processing tasks, but they do not specifically assess conversational capabilities. SD-Eval (Ao et al., 2024) claims to be an evaluation benchmark for spoken dialogue understanding and generation, evaluating the quality of response generation using automated metrics like BLEU, GPT-4o, or subjective evaluation. However, in this work, we specifically focus on the turn-taking capabilities of audio FMs, particularly their ability to understand, predict, and perform turn-taking events. In addition to previously proposed corpus-level statistics (Nguyen et al., 2023), we introduce our own metrics that evaluate the timing of turn-taking events. These metrics help us gain insights into the limitations of existing systems in engaging in interactive and natural conversations.

Method. 3 Turn-Taking Analyses based on Conventional Corpus-level Statistics In the given task setting, we evaluate the recently proposed dialogue systems with turn-taking capabilities using corpus-level statistics (Nguyen et al., 2023). The authors organize a user study and compute these statistics from the collected human-AI conversation data.

Human-human conversations contain a spontaneous succession of turns where overlap and silences occur naturally. Let us assume a two speaker conversation. For simplicity, we divide entire conversation audio into a sequence of N non-overlapping chunks U = {Ui | i = 1, . . . , N}. Then the voice-activity of the first speaker can be represented by Y1 = {y1 i |i = 1, ..N} where y1 i = 1 if speaker 1 is speaking in chunk i. Similarly, the voice-activity of speaker 2 can also be represented as Y2. We show the turn-taking events that occur in spontaneous spoken conversations in Fig. 1.

(1) Inter-Pausal Unit (IPU) is a continuous stretch of speech in one speaker’s channel, delimited by a silence of more than 200ms from both sides, such that IPUk = {i ∈[a, b]|yk i = 1}, where a and b denote the start and end chunk indexes of IPUk respectively and k can be 1 or 2. (2) Silence (S IL) is part of the conversation where there is no audio from both sides i.e. S IL = {i ∈ [asil, bsil]|y1 i = 0 and y2 i = 0}, where asil and bsil denote the start and end chunk indexes. (3) Pause is the silence that occurs between the successive IPUs of the same speaker. Successive IPUs by the same speaker separated by a pause are grouped to form a turn i.e. TURNk = {∪jIPUk j|[bj−1, a j] ⊂S IL}, where bj−1 is the end chunk index of IPUk j−1 and a j is the start chunk index of IPUk j. (4) Gap is the silence that occurs between IPUs of different speakers, i.e., it is the silence preceding a turn change. (5) Overlap is defined as the section of the conversation where both speakers are trying to speak While these metrics capture how well the global turn-taking events distribution in the generated dialogue matches with ground truth, it cannot evaluate the exact timing when a turn-taking event happens. For instance, Fig. 2 shows that Moshi can generate overlapping speech; however, it remains unclear whether this overlap is appropriate and supportive of the user’s statements or unexpected and disruptive to the natural flow of the conversation.

4 Proposed Turn-Taking analyses with Timing-Centric Metrics Inspired by the limitations of existing corpus-level statistics, we propose to evaluate the timing of the turn-taking event by training a model that predicts after every 40 ms chunk and is causal, i.e., trained to predict what turn-taking event will happen in the next 40 ms. We then use this predictor model as a proxy of human relevance judgments.

4.1 Judge turn-taking Model We take inspiration from prior work (Wang et al., 2024) that passes single-channel input speech through the SSL speech foundation model and predicts the listener’s behavior as turn-taking decisions that an ideal system should make. However, this approach cannot handle overlapping speech. Hence, we extend their labeling annotation scheme to model both the listener and speaker’s behavior and instead use single-channel mixed speech as input. We define the corresponding label sequence 4.2 Core Conversation Capabilities To develop metrics that comprehensively evaluate audio FMs’ conversational capabilities, we review prior literature (Gravano & Hirschberg, 2011; Skantze, 2021; Raux et al., 2006) and identify the key turn-taking abilities required for a conversational agent to interactively engage with an end user. We design metrics (Sec. 4.4-4.8) corresponding to each capability to ensure complete coverage of the skills required for natural human-AI dialogues. These capabilities are categorized depending on whether they are useful for user input recognition or system output generation, as shown below:

When user speaks : (a) When system should speak up? : This means that when the user speaks, the system should know when to listen, when to speak, and when to pause, ensuring that the conversation feels natural and not rushed.

Discussion. 6 Discussions and Conclusions Through an extensive survey of prior work, this work identifies the core turn management capabilities required to build an interactive voice assistant. To quantify the performance of existing audio FMs in these conversational aspects, we propose automated metrics corresponding to each capability, representing the first effort to assess the quality of turn-taking decisions made by audio FMs. Our findings offer valuable insights into the strengths and limitations of existing spoken dialogue systems. We plan to make our evaluation platform public, hoping that these proposed metrics will be adopted by future research to develop audio FMs that can engage in more natural and interactive conversations with users.

Limitations. 1 Scalability and Applicability: Our evaluation protocol needs a supervised dataset to train the judge model, limiting its scalability and applicability. This model can however be trained on any spoken conversation dataset containing speaker turns, transcripts, and timestamps, which are often available even for non-English languages as discussed in Sec. A.7. 2 Modelling BackChannels: Our current approach for identifying backchannels relies on heuristics (Sec. A.4) and may miss certain backchannels which is limitation of our model. 3 Performance of Turn Taking Model: The agreement between the judge labels and human judgments is below 80% for many metrics (Fig. 3). The primary contribution of this work lies in our novel and adaptable evaluation protocol, capable of integrating any turn-taking prediction model. Future efforts to improve model accuracy would further improve the protocol’s reliability. 4 Limitations of User Study: This study currently tests only a few audio FMs (Sec. A.9), and we plan to expand our benchmarking as more audio FMs with turn-taking capabilities emerge. There is the potential for bias in our user study, as participants were primarily the authors and their research colleagues (see Sec. A.2).