Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Abstract Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder [Cao et al., 2026] introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale.

Introduction. The human brain is organized into specialized functional systems that interact to support cognition [Fair et al., 2009]. Memory and reasoning likewise rely on partially distinct neural systems, allowing memory storage and cognitive computation to be functionally dissociated [Baddeley and Warrington, 1970, Squire, 2009]. In contrast, standard decoder-only language models [Singh et al., 2025, Team, 2026, Xu et al., 2026, Zeng et al., 2026] entangle long-term memory and reasoning within a single set of parameters. Long-term memory cannot be pretrained or scaled independently, and increasing memory size requires a corresponding increase in the total parameter count of the model. Under this shared parameterization, domain adaptation through continued pretraining or full finetuning requires optimization of the entire parameter set, incurs substantial training cost, and risks catastrophic forgetting [Kirkpatrick et al., 2017] or other unintended degradation of previously acquired capabilities.

Discussion / Conclusion. In this paper, we present Memory Decoder at Scale, scaling parametric memory models up to 6.9B parameters and pretraining them for 300B tokens. To enable memory pretraining at this scale, we developed a distributed Faiss pipeline based on embedding compression, index sharding, and parallel search, which addresses the indexing and search bottlenecks in constructing kNN distributions over 207B corpus tokens. Sparse kNN distribution storage further reduces space requirements, while distributed streaming loads only the entries required by each batch, enabling retrieval supervision construction at the scale of language model pretraining. Experiments across base model and memory scales reveal a consistent advantage for pairing small base models with large memory models. Scaling memory can therefore be more parameter-efficient than scaling the base model alone. A 6.9B general memory enables Pythia-410M to surpass Pythia-12B with 39% fewer total parameters.