When an AI remembers things, the real cost isn't storage — it's the computing needed to make old context usable.
What is the latency and compute cost of running memory inference?
This explores what it costs, in time and compute, for a model to use memory while it runs. That covers holding past context, retrieving it, and reconstructing it, not just storing it.
This explores what it costs, in time and compute, for a model to use memory while it runs. The corpus has no single latency figure for 'memory inference', but it does have a consistent answer about where the cost sits: mostly in the compute needed to turn old context into something usable, and in moving data around, rather than in storage. One paper reframes the long-context problem this way: the bottleneck is Is long-context bottleneck really about memory or compute?, not memory capacity. It describes consolidating evicted context into fast weights during offline 'sleep' phases. More consolidation passes give better results, especially on harder reasoning tasks, which is the same scaling pattern seen when models are given more thinking time. Remembering is something you pay for in compute, and you can turn that dial up or down.
How you retrieve memory changes the bill a lot, and a smarter design can be cheaper. MRAgent reconstructs memory on demand by walking a graph and pruning paths as evidence builds up, instead of running a fixed retrieve-then-reason pipeline. The corpus reports gains of up to 23% on reasoning tasks along with lower token and runtime cost (Can agents reconstruct memory on demand instead of retrieving it?). Forgetting can save time too. Most intermediate reasoning tokens stop mattering as a trace grows, so keeping only the instruction prefix and a recent window gives a 3x speedup with no training and lets traces run past 100k tokens (Can models think longer by forgetting intermediate reasoning?).
The cheapest memory may be one that skips the heavy computation. Engram pairs constant-time N-gram lookup with Mixture-of-Experts routing and beats a pure MoE at the same parameter count and FLOPs. The biggest gains show up in reasoning and code, not in plain retrieval (Can lookup memory and computation work together better than either alone?). On the hardware side, latency is often about data movement rather than arithmetic. MobileLLM found that on memory-bound phones, recomputing a transformer block twice is faster than fetching separate weights (Does recomputing weights cost less than moving them on mobile?). So counting FLOPs alone can mislead you about real response time.
At the low end, the numbers can be tiny. A 150M-parameter model that combines in-context examples with a recurrent latent memory reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, without writing out intermediate tokens (Can latent reasoning match chain-of-thought cost efficiency without verbalizing?). Deciding how much to spend is a separate question. The test-time scaling notes argue for adapting the budget to how hard the prompt is rather than using a fixed amount (Can we allocate inference compute based on prompt difficulty?). The same logic plausibly applies to memory: hard queries get deeper traversal or more consolidation, and easy ones get a cheap lookup. What the corpus doesn't have is a head-to-head benchmark of per-query memory latency across these approaches. It shows where the costs come from, but not yet a price list.
Sources 7 notes
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Most intermediate reasoning tokens become unimportant as reasoning progresses, so keeping only the instruction prefix and a recent window achieves 3x speedup without training while enabling traces beyond 100k tokens.
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
MobileLLM shows that on memory-bound mobile hardware, sharing weights between adjacent transformer blocks by recomputing one block twice uses less latency than fetching separate weights, gaining accuracy with no parameter increase.
Show all 7 sources
A 150M-parameter model combining in-context demonstrations with iterative latent computation reached 29.5% pass@2 on ARC-AGI-1 at $0.0007 per task, surpassing previously reported cost-accuracy tradeoffs. The approach separates learning (via demonstrations updating recurrent memory) from reasoning (via iteration in hidden space) without generating intermediate tokens.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Prefix Sliding for efficient test-time scaling
- Rethinking Thinking Tokens: LLMs as Improvement Operators
- Reasoning Models Can Be Effective Without Thinking
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases
- Memory is Reconstructed, Not Retrieved: Graph Memory for LLM Agents
- BDH-CQ: In-Context Learning with Recurrent Latent Reasoning
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models