Bolting a memory onto an AI can beat a much bigger model on some tests, but which kinds of tasks gain?
Which benchmarks benefit most from adding a separate memory module?
This explores which evaluation tasks see the biggest lift when a separate memory component is added to a model or agent, and the corpus has no ranked benchmark list, so the useful answer is about which kinds of task gain and which lose.
This explores which benchmarks gain most from a separate memory module. The corpus has no leaderboard for that, but it shows a pattern. Memory helps most where the bottleneck is stored knowledge or long-running state. It helps least, and sometimes hurts, where the test is whether a system genuinely learns from experience.
The clearest wins are knowledge-heavy and domain-adaptation tasks. A 6.9B memory paired with a 410M base model beat a 12B base-only model across 17 benchmarks with 39% fewer total parameters, and the result held across model scales and domain-adaptation tasks Does allocating parameters to memory beat scaling the base model?. The corpus reports that result in aggregate, so it doesn't say which of the 17 gained most. A dedicated memory model can also inject new knowledge into a frozen proprietary LLM without inference-time search that grows with corpus size. The cost is up-front training and limited capacity Can a separate memory model inject knowledge without touching the LLM?.
One result cuts against the obvious guess. When lookup memory (fast N-gram retrieval) was combined with mixture-of-experts computation, the gains were largest in reasoning and code, not in pure retrieval Can lookup memory and computation work together better than either alone?. You might expect recall benchmarks to benefit most, but here the memory frees up computation for harder thinking.
For long-horizon agent tasks, the evidence is suggestive but not clean. A four-level state cache, running from weights through context to disk-backed history, reported gains on ARC-AGI-3. The authors haven't ablated the individual components, so you can't credit the memory layer alone Can external state caches let models solve harder problems?. Terminal-Bench 2.1 shows a similar lift from an execution harness around frozen weights, including +5.4 points for DeepSeek-V4 Flash. That harness is a whole system, not a memory module in isolation Can execution harnesses lift model performance without retuning weights?.
The benchmarks where memory does worst are the ones that test whether it helps a system learn. On CL-BENCH, naive in-context learning beat dedicated memory architectures in most domains. The best system gained only 25% over a stateless baseline, and accumulated state introduced stale beliefs and spurious generalizations Do memory systems actually help language models learn continuously?. MemTrapBench is harsher. All five memory frameworks tested fell more than 10% below a no-memory baseline, even though the memories were accurate and relevant Can relevant memories actually harm LLM reasoning?.
These results are hard to compare because end-to-end scores hide where a memory system succeeds or fails. Evaluating storage, extraction, retrieval, and maintenance separately, as done across 12 systems, shows which component broke, not just whether the task passed How should we actually evaluate agent memory systems?. Before trusting a benchmark gain as a memory gain, check whether the task rewards stored knowledge, which memory tends to help, or learning from experience, where it often doesn't.
Sources 8 notes
A 6.9B memory paired with a 410M base model outperformed a 12B base-only model on 17 benchmarks while using 39% fewer total parameters. This result held across multiple model scales and domain-adaptation tasks, suggesting memory is a more efficient capacity axis than base-model growth.
MeMo trains a dedicated memory model to encode new knowledge, eliminating inference-time search costs that scale with corpus size. It avoids fine-tuning risks and works with frozen proprietary models, but trades this for up-front training cost and capacity limits.
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
Prime Agent organizes persistent state in four levels (weights, context, persistent REPL plus subagents, disk-backed history) to let models read and write addressable state beyond their instruction stream. The approach isolates harness failures from model failures and reported gains on ARC-AGI-3, though specific components remain unablated.
StateM improves Terminal-Bench 2.1 accuracy across multiple models by optimizing execution systems around frozen weights. The same runbook transfers to newer models without modification, achieving 95.3% on GPT-5.6 and lifting DeepSeek-V4 Flash by 5.4 points.
Show all 8 sources
CL-BENCH's gain metric isolates true learning from base capability and finds that naive in-context learning outperforms dedicated memory architectures on most domains, with the best system gaining only 25% over a stateless baseline. Accumulated state introduces spurious generalizations and stale beliefs.
MemTrapBench shows that all five tested memory frameworks underperform a no-memory baseline, with drops exceeding 10%, despite memories being accurately stored and task-relevant. This reveals a failure mode at the point of use that standard memory benchmarks miss.
Decomposing memory into storage, extraction, retrieval, and maintenance stages exposes design trade-offs and failure modes that task-success metrics completely hide. Module-by-module evaluation across 12 systems shows which component actually failed rather than just whether the task succeeded.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
- GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agents
- Provable Benefits of In-Tool Learning for Large Language Models
- From Model Scaling to System Scaling: Scaling the Harness in Agentic AI
- Are We Ready For An Agent-Native Memory System?
- SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs
- Continual Learning Mechanisms Compose for Long-Horizon Memorization