Does allocating parameters to memory beat scaling the base model?

Memory Decoder at Scale takes the parametric long-term memory module from the original Memory Decoder work and scales it to 6.9B parameters, pretrained on 300B tokens. Its central finding is that "across model scales, allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone." The headline case: on 17 benchmarks, a 6.9B general memory paired with Pythia-410M raises the average score from 29.86 to 37.34, which passes Pythia-12B (37.24) with 39% fewer total parameters. The domain-adaptation result points the same way: for Qwen3 Base models from 0.6B to 14B, a 1.7B domain memory improves the three-domain average by more than 9 points at every scale.

The paper's framing is that decoder-only models "entangle long-term memory and reasoning in a single parameter set," so memory cannot be pretrained or scaled on its own, and adding memory means adding total parameters to a model that must also be optimized as a whole. Separating the two makes memory a second axis of capacity. The brain analogy in the introduction (memory and reasoning rely on partially distinct neural systems) is offered as motivation, not as evidence. The same separation is the argument for domain adaptation: a memory can be swapped or trained without touching the base model's weights, which avoids the training cost and forgetting risk the introduction attributes to continued pretraining and full finetuning.

Much of the paper's engineering effort goes to making the memory supervision constructible at all. The training signal is kNN distributions over 207B corpus tokens, and at that size "the combined cost of indexing and search makes a standard Faiss pipeline infeasible." The fix is a distributed Faiss pipeline (embedding compression, index sharding, parallel search) plus sparse storage of kNN distributions that streams in only the entries each batch needs. So the scaling result depends on a data-pipeline advance as much as on the model.

Against the neighbors, this note adds a scaling axis. Can retrieval knowledge compress into a tiny parametric model? showed that retrieval behavior can be compressed into a small decoder; this paper asks what happens when that decoder grows, and answers that memory parameters buy more than base-model parameters. The parameter-allocation framing parallels Can lookup memory and computation work together better than either alone?, where capacity split between lookup and experts beats spending it all on experts. There the memory sits inside the model; here it is a separate pretrained module. It also complements Can a separate memory model inject knowledge without touching the LLM?, which decouples corpus growth from inference cost, whereas this paper decouples memory capacity from base-model size.

The excerpt does not say which of the 17 benchmarks drive the gain, so it cannot show whether the advantage is concentrated in knowledge-heavy tasks or spread evenly. It also counts total parameters only. It says nothing on latency, compute per token, or the cost of running memory and base model side by side, and it gives no numbers for scales in between or beyond those tested. Read at the strength given, the result is that in this setup, at the scales tested, a small base plus a large memory is the more parameter-efficient way to reach a given score. It does not show that memory scaling wins on every cost measure or that it substitutes for a larger base model on reasoning-heavy work.