VoiceMem: Streaming Dual-Brain Memory for Real-Time Interaction

Conversational systems, such as duplex speech language models (SLMs), still lack a streaming, accurate, and empathetic memory system as their soul. We introduce VOICEMEM, a simple memory architecture with a parallel informational left brain, an emotional right brain, and streaming memory I/O mechanisms. We further build a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment with interchangeable memory backends. Experiments and real-world deployment show three advantages: i) Accuracy: under top-5 retrieval, the left brain outperforms classical systems such as MEM0 at top-200 by nearly 30 points; ii) Emotional & Personal: the right brain, with short- and long-horizon affective attribution and dual-node persona modeling, achieves stateof-the-art performance across three persona benchmarks and improves the aggregate score by 1.89 points over the previous best system; and iii) Real-Time & Cheap: VOICEMEM completes retrieval in 134 ms, well within standard VAD latency, adding no extra conversational delay while maintaining high accuracy and low cost. These results show that VOICEMEM provides a practical memory foundation for real-time, personalized, and emotionally aware speech interaction.

Introduction. "Through memory, the soul reveals traces of its former existence." — Plato Memory is what turns a conversational system (e.g. voice agent) from an intelligent tool into a humancentered partner. Although many powerful memory systems have appeared recently (Chhikara et al., 2025; Xu et al., 2025b; Hu et al., 2026), alongside interaction models that are increasingly intelligent and natural (e.g. SLMs (Xie and Wu, 2024a; Xie et al., 2026b; Fang et al., 2025; Qwen Team, 2026a) and duplex models (Ma et al., 2025; ByteDance Seed, 2026; OpenAI, 2026)), the two have not been merged into one complete solution. Building an empathetic and intelligent interaction system on top of memory remains an open challenge. From our view, three obstacles stand in the way. (O1) Unified architecture for both informational and emotional intelligence. For real-time conversational systems, a grasp of emotion and persona matters as much as information and intelligence, and the mechanisms behind emotion are far more complex. How to model both in one architecture over the long run is still unexplored.

Discussion / Conclusion. In this paper, we presented VOICEMEM, a streaming dual-brain memory framework that equips real-time conversational systems with both informational and emotional memory without breaking their latency budget. The left brain organizes factual knowledge through a two-level schema–entity index with a query-driven cluster emergence mechanism, keeping the candidate pool dense enough to survive a top-5 retrieval budget, while the right brain models the person through independent and cross-entity persona nodes maintained by short- and long-horizon affective attribution. A four-stage streaming query then hides the entire retrieval inside the silence that a standard VAD already waits out. Around this core we built a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment with interchangeable backends.