Two ways to make AI cheaper to run: look answers up, or only use part of the model. Which wins?
How does conditional memory compare to conditional computation for efficiency?
This explores whether looking things up (conditional memory) and only running part of the network per token (conditional computation, as in Mixture-of-Experts) are rival ways to save effort, or two separate efficiency levers.
This explores whether looking things up (conditional memory) and only running part of the network per token (conditional computation, as in Mixture-of-Experts) are rival ways to save effort, or two separate levers. The corpus points to two levers. Engram pairs an O(1) N-gram lookup table with MoE routing and finds a U-shaped scaling law: for a fixed parameter and compute budget, the best results come from splitting capacity between the two, not from putting it all into experts or all into lookup. The hybrid beats pure MoE at equal parameters and FLOPs, so Can lookup memory and computation work together better than either alone? treats memory as a second sparsity axis next to computation.
The gains show up mostly in reasoning and code, not in pure fact retrieval. You might expect a lookup table to help most with recall. It fits a different picture: if the network doesn't have to spend layers rebuilding common patterns, that compute is available for harder steps. That is my reading of the result, not something the note states outright.
The corpus also shows that memory isn't free, because the cost moves somewhere else. Is long-context bottleneck really about memory or compute? argues the long-context bottleneck is the compute needed to turn old context into usable internal state, not storage capacity. A separate memory model, as in Can a separate memory model inject knowledge without touching the LLM?, makes inference cost independent of corpus size. In exchange it costs more training up front and has a capacity ceiling. That ceiling is real: Can state-space models match transformers at copying and retrieval? shows a fixed-size memory state cannot copy or retrieve as well as a transformer that can look back at the full context. Compressed memory saves compute but loses exact recall. Can neural memory modules scale language models beyond attention limits? tries to soften this by storing only surprising tokens.
A third family makes both memory and compute conditional at run time, deciding what to keep and how much to think. Can models think longer by forgetting intermediate reasoning? finds most intermediate reasoning tokens stop mattering, so keeping only the prompt prefix plus a recent window gives a 3x speedup with no training. Can recursive subtask trees overcome context window limits? and Can reasoning systems forget history without losing coherence? push the same idea by pruning cache or discarding history. Can agents reconstruct memory on demand instead of retrieving it? rebuilds memory by traversing a graph on demand and cuts token cost. Can we allocate inference compute based on prompt difficulty? does the compute-side version, giving easy prompts less budget and hard ones more.
There are two limits on these savings. Skipping compute does not replace training: Can non-reasoning models catch up with more compute? finds that extra inference budget can't close the gap with a model trained to reason. And lookup only saves effort if what comes back helps. Can relevant memories actually harm LLM reasoning? found that all five tested memory frameworks scored below a no-memory baseline, even with accurate, relevant memories. The corpus has only one paper that compares the two axes directly (Engram), so the broader story here is assembled from adjacent work on caching, pruning and adaptive budgets.
Sources 12 notes
Engram combines O(1) N-gram lookup with Mixture-of-Experts routing, revealing a U-shaped scaling law where balanced allocation to both mechanisms outperforms either alone. Gains appear largest in reasoning and code rather than pure retrieval.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
MeMo trains a dedicated memory model to encode new knowledge, eliminating inference-time search costs that scale with corpus size. It avoids fine-tuning risks and works with frozen proprietary models, but trades this for up-front training cost and capacity limits.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
Show all 12 sources
Most intermediate reasoning tokens become unimportant as reasoning progresses, so keeping only the instruction prefix and a recent window achieves 3x speedup without training while enabling traces beyond 100k tokens.
The Thread Inference Model demonstrates that reasoning structured as recursive subtask trees with rule-based KV cache pruning sustains accurate reasoning beyond context limits, even when manipulating 90% of the cache. This enables single models to replace multi-agent systems by handling full recursive reasoning internally.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
MemTrapBench shows that all five tested memory frameworks underperform a no-memory baseline, with drops exceeding 10%, despite memories being accurately stored and task-relevant. This reveals a failure mode at the point of use that standard memory benchmarks miss.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- Reasoning Models Can Be Effective Without Thinking
- Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets
- Repeat After Me: Transformers are Better than State Space Models at Copying
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Titans: Learning to Memorize at Test Time