Toward Enabling Natural Conversation with Older Adults via the Design of LLM-Powered Voice Agents that Support Interruptions and Backchannels

Source: Chao Liu, Mingyang Su, Yan Xiang, Yuru Huang, Yiqian Yang, Kang Zhang, Mingming Fan; CHI 2025 full paper · 2025-04

Voice agents can construct meaningful conversations with older adults to offer various benefits, such as providing emotional companionship and assisting with memory recall. However, such conversations often follow the simple turn-taking pattern and lack interruption and backchannel of natural human conversation. Previous research has shown that this rigid turn-taking pattern lacks interactivity and initiative, limiting the flexible communication between older adults and voice agents. To address these issues and create a more natural conversational voice agent, we first conducted a formative study to identify common usage of interruption in the natural conversations of older adults. We then designed an LLM-powered Barge-in agent that supports interruption and backchannel. Our within-subject exploratory study showed that participants felt that conversations with Barge-in agents were more natural, engaging, and fluent than with the No barge-in agent. We further present design implications for creating more natural and human-like voice agents for older adults.

Introduction. Social isolation and loneliness are major risk factors for many older adults that negatively affect both physical and mental health, leading to conditions like depression, reduced quality of life, and even increased mortality. With the continuous progress of voice user interfaces, some voice agents designed for older adults are emerging. For example, Alexa’s Ask My Buddy and Google Assistant’s Vigil Connect, both developed based on traditional language models, can provide certain levels of emotional companionship help for older adults. Compared with text conversation, older adults prefer direct voice interaction for conversations because it is more intuitive than touchscreens and keyboards, requiring neither precise hand-eye coordination nor extensive learning. This ease of use also makes it simpler for older adults to access information and services through voice agents. Prior to the advent of large language models (LLMs), academic research on voice agents designed for older adults primarily focused on task-oriented interactions, such as information search, reminders, and other specific tasks. Conversational companionship was typically a secondary function, as these traditional voice agents relied on limited language models that lacked the complexity and depth needed for more dynamic, human-like interactions. With the rapid development of LLMs, such as OpenAI’s ChatGPT, agents have gained significant improvements in the ability to generate contextually aware and natural conversations. These LLM-powered agents have narrowed the gap between human language and machine-generated responses, resulting in more fluid, natural interactions. Furthermore, multimodal models like OpenAI’s Whisper, Google’s AudioPaLM, and Microsoft’s Azure Cognitive Services have enabled voice agents to produce speech outputs with richer emotional and rhythmic tones. These advances present new opportunities for creating voice agents tailored to the needs of older adults, which may be designed to offer more personalized and socially aware interactions. Despite significant technical advancements in natural language processing, there are still notable challenges in developing voice agents that effectively engage older adults. One of the more critical issues is that current voice agents still primarily rely on rigid turn-taking protocols, commonly called the "speak-wait/speak-wait" mode, in which one party speaks while the other waits. This linear approach contrasts sharply with natural human-human conversations, which often involve interruptions and backchannels that maintain the flow of dialogue. The traditional voice agent presents four notable limitations for older adults. Firstly, during user turns, it necessitates the construction of complete sentences and full organization of thoughts prior to speaking. However, older adults often prefer utilizing shorter, less complex sentences and frequently rely on ambiguous expressions. Secondly, during the voice agent’s turns, the decline in hearing and communication abilities among many older adults hinders their effective comprehension of complex conversations. Consequently, they often require questions or repetitions to ensure accurate information grasp. Nevertheless, the model typically employs a simplex communication protocol, which is a one-way communication method where the system only receives input from the user without permitting real-time interruptions or corrections, thereby impeding this function. Thirdly, older adults require more time to adapt and are prone to making mistakes when using traditional turn-taking agents. Their tendency to speak more slowly can lead to the system incorrectly assuming the end of input during pauses within a sentence, resulting in incomplete or failed conversations. Moreover, the inability to interrupt or correct the voice agent in a timely manner can cause frustration. Lastly, current voice agents in the voice modal commonly lack backchannels, leading to a deficiency in interactivity and sociability. Additionally, these systems frequently utilize simple Voice Activity Detection (VAD) to detect the conclusion of a user’s speech, and when the user remains silent, the voice agent does not initiate further conversation. This silence can be more readily triggered in older adults compared to younger individuals due to their unfamiliarity with the system or incorrect usage. Furthermore, while some products like Hume’s EVI and Moshi AI allow users to interrupt agents, these interruptions remain largely one-sided (only the user interrupts the machine), with no mechanism for the voice agent to initiate or manage interruptions, limiting the interaction’s naturalness and fluidity.

Related work. Most voice agents currently follow a strict alternating dialogue model, also known as turn-taking protocol, where one party speaks while the other waits. This model relies on techniques like wait time, syntax analysis, and context understanding to detect the end of a turn, signaling when the system should respond. However, human-to-human conversations are far more dynamic, often involving interruptions and backchannels that contribute to the fluidity and naturalness of the interaction, rather than strictly adhering to turn-taking protocols.

2.1.1 Interruptions. In conversation, an interruption is defined as an intentional act by one conversational participant to break into another’s speech. This typically occurs at a non-transition-relevance place, where the speaker has not yet completed their turn or reached a natural transition point. Previous research has shown that interruptions play a critical role in human-computer interactions. Matsusaka et al. designed a multimodal agent capable of monitoring multi-user dialogues, interrupting when corrective information needs to be provided. However, they did not explore the user experience in this context. Similarly, Palinko et al. demonstrated that using a prediction-based interruption model, combined with non-verbal signals, improved the user acceptance of robotic participation in multi-person conversations. Despite these advances, the design of interruptions for single-user and robot conversations remains a critical challenge. Interruptions themselves have been classified into several types based on their function and impact on the conversation. Early studies, such as those by Beattie et al., categorized interruptions by turn transition intensity, distinguishing between types like silent interruptions and butting-in interruptions. Later, Roger et al. differentiated between interruptive speech (e.g., intrusive interruptions that disrupt the conversation) and non-interruptive speech (e.g., supportive overlaps aimed at assisting the speaker). Murata et al. further refined this classification, categorizing interruptions based on their supportive or competitive nature into co-operative interruptions and competitive/intrusive interruptions. The latter category is further subdivided into three types: topic-changing interruptions, floor-taking interruptions, and disagreement interruptions. However, human-human conversational behavior changes with age, and there is a lack of research on older adults’ preferences for these different types of interruptions.

Method. To answer the RQ1, we conducted a formative study where older adults (N= 8) engaged in human-to-human conversations, retrospective interviews, and interactions with a turn-taking LLM agent. The goal of the study was to identify key DCs for developing a voice agent that effectively supports interruptions and backchannels for older adults. We focused on three key questions: What types of interruptions do older adults commonly use in in-person conversations? How do older adults perceive interruptions and backchannels in human-to-human conversations? And what are their expectations for agents that can support such behaviors? We gathered participants’ feedback through retrospective think-aloud sessions. All studies received ethical approval from our institution. Participants could withdraw at any time, but no participants opted to withdraw.

Participants were asked to engage in conversations on two carefully selected topics, with the goal of helping them quickly enter a conversational state, which would facilitate the recording of interruptions by the researchers. The first topic involved discussing photos, given that research has indicated the significant role as a tool for stimulating the interest of older adults in conversational contexts. Before the study began, we asked each participant to select three personal photos they were most eager to share (6 photos per pair). Given the vulnerability of older adults, we carefully considered ethical aspects and conducted a preliminary screening of all submitted photos to ensure that no participant would feel uncomfortable. Existing research suggests that conversations among acquaintances often involve more emotional and experiential sharing. Therefore, the second topic centered on recalling shared positive experiences.

The data collected during the formative study consists of three parts: 1) video recordings of each participant pair’s human-to-human conversations; 2) feedback from participants on their perceptions of being interrupted and interrupting others, collected through retrospective think-aloud sessions; 3) full recordings of all one-on-one semi-structured interviews. The video recordings were transcribed using a commercial automatic speech recognition (ASR) system, iFlyrec, with the transcriptions manually verified for accuracy by the research team. We adopted Okamoto et al.’s more contextual measurement to determine whether a speech act constitutes an interruption, considering situational factors such as the current speaker’s intentions and the content of both speakers’ utterances, rather than relying solely on syllables. To track interruptions and backchannel behaviors during both the formative and user studies, we developed a Python plug-in that allowed for efficient timestamp marking and video retrieval for detailed review. For the data analysis, we followed these steps: Four researchers independently coded the transcripts using the open coding method, supported by FigJam board. During the coding process, we identified and defined multiple codes. Thinking Aloud records were used as a supplemental data source to validate and further elaborate on the open codes derived from the transcriptions. By analyzing participants’ behaviors related to interrupting others and being interrupted, we gained a deeper understanding of their perspectives on interruptions and their expectations for voice agent interactions. This analysis helped to confirm and refine our initial coding. This research method has been shown to be effective in studies related to interaction and conversational dialogue with older adults. Selected quotes from the phase, originally in Chinese, were translated by the first author and subsequently reviewed by co-authors to ensure accuracy.

DC1- The voice agent should prioritize co-operative interruptions when interacting with older adults, while incorporating a diverse range of interruption types. Murata et al. classified interruptions into two main types: Co-operative Interruption and Competitive/Intrusive Interruption.

Discussion. In this section, we explore insights derived from evaluating the LLM-powered Barge-in Agent for older adults, providing guidance for the future design of voice agents that simulate natural conversations for this demographic. Our discussion focuses on three key aspects: proactively guiding topics through contextual cues and timely interruptions, tailoring interruption and backchannel strategies to accommodate older adults’ sensory perceptions, and optimizing conversational coherence based on cognitive feedback and adapting interruption strategies. By addressing these factors, we highlight how future voice agents can enhance engagement and fluency compared to traditional voice dialogue systems.

7.1 Guiding Topics Proactively Through Contextual Cues and Timely Interruptions 7.1.1 Leveraging personal context to enhance topic relevance. In conversations with older adults, topics are particularly important, especially in maintaining engagement and enhancing the sense of participation. Before the study, we asked participants about their daily lives and hobbies to ensure that the conversation would revolve around topics they were familiar with and interested in. Previous studies have also shown that starting a conversation with familiar topics helps improve its continuity and flow. Most participants indicated that this made them feel more comfortable and gave them a sense that the agent understood them, which in turn enhanced their sense of participation. Participants generally did not mind that this personal information was being collected by the agent, as they believed that sharing such information was natural and necessary, even in everyday communication.

7.1.2 Timely topic-shifting interruptions guided by contextual relevance. By introducing timely interruptions, whether agent-initiated or participant-initiated, the conversation can be effectively guided in new directions, making it more diverse and enriching. In our study, the agent used topic-changing interruptions to introduce new topics, successfully broadening the scope of the discussion. As P6 said, "The agent’s interruptions introduce new topics, making the conversation more diverse," which participants found more engaging compared to a No Barge-in Agent. At the same time, when the topic deviates from the participant’s interests, participants may also interrupt to steer the conversation toward topics they prefer. Participants generally want the conversation to revolve around topics they care about and strive to keep the conversation interesting.

7.2 Tailoring Interruption and Backchannel Strategies to Older Adults’ Sensory Perceptions 7.2.1 Mitigating auditory perception variability with multimodal feedback. In our study, we observed differences in participants’ perceptions of interruptions and backchannel signals. Previous studies have also indicated that with advancing age, older adults face not only the issue of hearing loss but also changes in the central auditory system, including differences in processing times, alterations in neural response patterns, and a slowing down in the response speed to rapid sequences of sound events. Some participants reported that the backchannel (e.g., "yes," "uh-huh") and active interruptions made by the Barge-in Agent interfered with their speech, this is likely because the auditory system struggles to quickly process and distinguish between these rapid auditory cues, making it difficult for them to seamlessly integrate these signals into ongoing conversations without feeling disrupted. In addition, excessive backchannel responses can lower users’ evaluations of the robot, which is consistent with observations in the field of voice chat.

Conclusion. Our research highlights the importance of incorporating interruption and backchannel capabilities into voice agents to enhance interactions with older adults. Our LLM-powered Barge-in Agent provides a more natural and engaging conversational experience. The formative study revealed the frequent use of interruptions and backchannels in older adults’ natural conversations, shaping the design of our agent. The within-subject exploratory study showed that most participants found conversations with the Barge-in Agent more engaging and fluid compared to a standard turn-taking agent. These findings suggest that integrating human-like conversational features is essential for agents aimed at providing emotional companionship and supporting memory recall in older adults. We hope that our design insights will inspire the development of more natural, human-like agents, ultimately improving the quality of life for older adult users.

Limitations. While our study provides insights into improving interactions between older adults and voice agents, several limitations suggest areas for further investigation. The voice agent in this study focused on emotional companionship, but its applicability to more task-oriented agents is unclear. To collect data on interruptions and turn-taking, we selected couples and same-age friends as participants. However, interactions with younger family members or different relationship types may exhibit different patterns, affecting the generalizability of our findings. Future research should examine how older adults interact with various demographic groups (e.g., children, grandchildren) to inform agent design. Although we allowed customizable parameters, such as response wait times and interruption frequencies, uncertainty in network latency and token calculation speed during the study prevented us from fully exploring their effects. Future studies should investigate these parameters and offer adaptive settings to better meet individual needs.