A voice assistant can look things up in the pauses between your sentences, so the harder question is what's worth remembering.
When should voice agents write new memories versus read existing ones?
This explores the timing question for a voice agent: during a live conversation, when is it worth saving something new, and when should it look up what it already knows?
This reads the question as a timing problem for live conversation. The corpus has nothing on when a voice agent should write memories, so this answer combines the one voice-specific retrieval note with general agent-memory work. Reading turns out to be nearly free in voice, so the real question is what deserves a write and when to make it.
Start with reading. VoiceMem does retrieval in 134 ms, inside the silent gaps that voice-activity detection already leaves between turns, and it runs a factual branch and an emotional branch in parallel (Can memory retrieval hide inside voice agent silence?). Latency, the usual reason to read sparingly, mostly goes away. Quality is still a limit. Retrieved memory helps or hurts depending on its role: clarifying memory improves accuracy, while irrelevant memory actively degrades it (Does retrieved memory quality depend on its functional role?). So the discipline for reading is filtering by role, not rationing. Reading is also where the thinking can happen. MRAgent stores memory as a graph and reasons over it while traversing it, which beat retrieve-then-reason by up to 23% at lower token cost (Can agents reconstruct memory on demand instead of retrieving it?). That suggests writes can stay simple and the heavy work can wait for read time.
Writes need a trigger worth trusting. Reflexion shows the cleanest one: an unambiguous success or failure signal lets an agent write a useful self-diagnosis without rationalizing, and leaving that note uncompressed keeps it usable (verbal-reflection-stored-as-episodic-memory-lets-agent-learn-from-trial-and-err). In a voice setting the closest analogues are a user correcting the agent or stating a preference outright. That mapping is my inference, not something the note tests. The second trigger is a pause. DeepAgent folds long interaction history into episodic, working, and tool memory when the agent stops to reconsider, and that mix of autonomy and structure avoids the degradation of sloppy consolidation (Can agents compress their own memory without losing critical details?). RAISE adds that memory operates at two scales: dialogue-level pieces like history and scratchpad, and turn-level pieces like examples and task trajectory. Each has its own failure modes and update policy, so "write now" means different things at each scale (How should agent memory split across time scales?).
Then there is who makes the call. The corpus describes an explicit hot path, where the agent decides mid-conversation through a tool call, and an implicit background path triggered programmatically. Each trades context-sensitivity against reliability differently across writing, storing, retrieving, and deleting (How should agents decide what memories to keep?). Putting this together, and this too is my inference, a voice agent could read every turn in the silent gaps. It could hand routine writes to a background process after the turn or session, and keep hot-path writes for moments like an explicit "remember this" or a correction.
How a memory is written matters as much as whether it is written. Copying a large agent's memories into a small one gave only marginal gains, but restructuring them into workflow, subtask, and function memory closed much of the gap, up to 27.2 points (Can small agents learn from larger agents' memories?). AgentFly reached 87.88% on GAIA by improving through memory operations alone, with no weight updates (Can agents learn continuously from experience without updating weights?). So each write is a learning event, and a voice agent should make fewer, better-structured writes rather than log everything.
Sources 9 notes
VoiceMem splits memory into parallel informational and emotional branches, completing retrieval in 134 ms inside existing VAD gaps. The system outperforms competitors on factual retrieval and persona benchmarks while adding no conversational latency.
Retrieved memory type drives response quality more than relevance alone: clarifying memory improves factual accuracy and constraint awareness, while irrelevant memory actively degrades both. Role-aware retrieval and filtering are robustness requirements, not optional optimizations.
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.
DeepAgent's autonomous memory folding consolidates interaction history into episodic, working, and tool memory schemas. This reduces token overhead while letting agents pause to reconsider strategies—the autonomy and structure together avoid degradation that plagues poorly designed consolidation.
Show all 9 sources
RAISE shows that agent memory consists of four components organized by two design axes: dialogue-level (conversation history, scratchpad) versus turn-level (examples, task trajectory). This granularity distinction predicts different failure modes and update policies for each component.
Memory management decomposes into explicit hot-path (agent decides via tool calling) and implicit background (programmatically triggered) paths. Each approach trades context-sensitivity for reliability differently across generation, storage, retrieval, and deletion.
Naive memory transfer from large to small agents yields only marginal gains, but restructuring the teacher's trajectories into workflow, subtask, and function memory—without retraining—closes the gap significantly. Students of 4B to 8B parameters achieved up to 27.2 percentage point gains and sometimes approached teacher-level performance.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory
- PGMem: Tightly Coupled Persona-Memory Graph for Lifelong Personalized Agents
- ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory
- Rethinking Memory as Continuously Evolving Connectivity
- Are We Ready For An Agent-Native Memory System?
- Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents