Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory

Memory systems have shown promise for improving agent performance, but their potential remains largely unexplored for small language models, which struggle to generate sufficient successful trajectories on their own. We propose Agent Memory Distillation (AMD), a training-free framework that transfers structured knowledge from a large teacher agent to a small student agent through hierarchical memory. AMD constructs three complementary memory types from successful teacher trajectories: Workflow memory encodes task-level strategies, Subtask memory provides concrete behavioral examples at an intermediate granularity, and Function memory captures perfunction calling conventions and common pitfalls. Workflow and Subtask memories are injected proactively at the start of each task, while Function memory is retrieved reactively upon tool-calling errors. We evaluate AMD on three tool-use benchmarks using four student models (4B–8B parameters) with GPT-5- mini as the teacher, achieving average accuracy gains of 27.2%p, 11.2%p, and 3.4%p on AppWorld, BFCL V3, and ToolSandbox, while consistently outperforming existing memorybased baselines.

Introduction. Memory has emerged as a critical component for developing capable self-evolving agents, enabling the reuse of successful behavioral patterns and the avoidance of past failures (Zhao et al., 2024a; Ouyang et al., 2025; Zhang et al., 2026). As modern agents increasingly rely on external tools such as APIs and predefined functions to tackle complex tasks, the role of memory becomes even more essential (Liao et al., 2025; Xu et al., 2026). In such settings, memory helps agents recall effective tooluse strategies from past interactions, improving both efficiency and task success rates (Fang et al., 2025; Xia et al., 2025). Moreover, memory can encode tool conventions such as argument schemas and return structures, enabling agents to invoke tools more accurately and reliably (Du, 2026). However, the potential of memory remains largely unexplored for small agents, in contrast to its demonstrated effectiveness with large propri- etary models (Wu et al., 2025; Luo et al., 2026).

Discussion / Conclusion. We presented Agent Memory Distillation (AMD), a training-free framework for transferring teacher agent experiences to small student agents through hierarchically structured memory. We first identified that naive memory transfer yields only marginal improvements due to the capability gap between teacher and student agents, and showed that this gap cannot be bridged by simply providing high-quality teacher memories. AMD addresses this by constructing three complementary memory types at different levels of task granularity: Workflow memory for high-level planning, Subtask memory for concrete behavioral references, and Function memory for fine-grained tool invocation guidance. Experiments across three benchmarks with four student models demonstrate that AMD consistently outperforms zero-shot baselines and all memory-based baselines, with some students approaching or even surpassing teacherlevel performance. Ablation studies further confirm that each memory type contributes distinctly to knowledge transfer, and that effective distillation requires careful alignment between memory complexity and student capacity.