Should an AI's working memory during one task and its long-term saved memory follow the same rules?
How do inference lifecycle needs differ from memory fragment governance?
This reads the question as asking how managing what a model holds in mind while it works through a single task differs from managing the stored pieces of memory an agent keeps across tasks: what gets saved, kept, rebuilt or thrown away.
This reads the question as a contrast between two memory problems: what a model needs to keep in view while it reasons through one task, and how an agent should manage the stored bits of memory it carries from one task to the next. The corpus covers the first problem far better than the second. It has a lot on inference-time memory and very little on governance in the policy sense: who owns a stored memory, when it should expire, how it can be audited or deleted. If governance is what you came for, the collection has a real gap there. What it does show is that the two problems pull in opposite directions.
During a single inference, the big finding is that most of what a model writes while thinking is disposable. Intermediate reasoning tokens quickly stop mattering. Keeping only the original instruction and a recent window of tokens gives roughly a 3x speedup and allows reasoning traces past 100k tokens Can models think longer by forgetting intermediate reasoning?. Atom of Thoughts goes further: each step depends only on the current state of the problem, not on the history that led there Can reasoning systems forget history without losing coherence?. The Thread Inference Model splits reasoning into nested subtasks and prunes the model's cache of past tokens once each subtask finishes, so the model keeps working accurately long after its context window would normally run out Can recursive subtask trees overcome context window limits?. At inference time, the goal is to forget aggressively and on purpose. The same idea appears in system design. LLM Programs give each model call only the context its step needs Can algorithms control LLM reasoning better than LLMs alone?, and ReWOO-style planning keeps tool outputs from piling up in the prompt Can reasoning and tool execution be truly decoupled?.
Stored memory reverses that instinct. The question is no longer what you can drop but how stored fragments should be organized, reached and kept consistent. The surprising move here is that some researchers stop treating memory as a filing cabinet. MRAgent rebuilds memory on demand by walking a graph and reasoning along the way, instead of fetching fixed chunks and reasoning afterward. The relationships between memories are worked out when they're needed, not stored in advance Can agents reconstruct memory on demand instead of retrieving it?. Others argue that memory shouldn't sit in a separate module at all. Metis builds memory state and memory procedures into the model itself, because an external store and the model, trained separately, drift apart Should agent memory live inside the model backbone?. MeMo takes the opposite route: a dedicated memory model that leaves the main LLM untouched, at the price of fixed capacity Can a separate memory model inject knowledge without touching the LLM?.
The bridge between the two problems is the most interesting finding. One line of work argues that long-context limits aren't really about storage space. They're about the compute it takes to turn context that has fallen out of the window into lasting internal knowledge, through offline "sleep" passes that improve the more you run them Is long-context bottleneck really about memory or compute?. Seen this way, the throwaway working memory of inference is the raw material for long-term memory, and consolidation is the step that decides what survives. That decision point is where governance questions like provenance, correction and forgetting would naturally belong. The corpus describes how the step works but says almost nothing about who should control it.
Sources 9 notes
Most intermediate reasoning tokens become unimportant as reasoning progresses, so keeping only the instruction prefix and a recent window achieves 3x speedup without training while enabling traces beyond 100k tokens.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
The Thread Inference Model demonstrates that reasoning structured as recursive subtask trees with rule-based KV cache pruning sustains accurate reasoning beyond context limits, even when manipulating 90% of the cache. This enables single models to replace multi-agent systems by handling full recursive reasoning internally.
LLM Programs embed LLMs within explicit algorithms that manage control flow and state, presenting only step-specific context to each LLM call. This information hiding addresses capability and context window limits while treating complex reasoning as modular, debuggable sub-tasks.
ReWOO and Chain-of-Abstraction both decouple reasoning from tool responses through different mechanisms—planning-before-execution and abstract placeholders respectively—eliminating quadratic prompt growth and sequential latency while maintaining reasoning quality.
Show all 9 sources
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Metis demonstrates that agent memory can be implemented as a persistent state and autonomous procedures within the model backbone rather than external modules. This approach enables end-to-end training and avoids the decoupling failures where external memory and backbone optimize independently.
MeMo trains a dedicated memory model to encode new knowledge, eliminating inference-time search costs that scale with corpus size. It avoids fine-tuning risks and works with frozen proprietary models, but trades this for up-front training cost and capacity limits.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Know It, Act on It: Investigating Memory Utilization in LLM Personalization
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity
- Prefix Sliding for efficient test-time scaling
- MemTrapBench: Benchmarking Cognitive Traps in LLM Memory Use
- Efficient Tool Use with Chain-of-Abstraction Reasoning
- Are We Ready For An Agent-Native Memory System?
- Useful Memories Become Faulty When Continuously Updated by LLMs
- Agent Memory Distillation: Empowering Small LLM Agents with Hierarchical Teacher Memory