When does active reconstruction cost more than simple context dumping?
This explores the tradeoff between *rebuilding* what you need on the fly — traversing a memory graph, consolidating context into internal state, reasoning your way back to relevant facts — versus just *handing the model the raw text* and letting it read. The corpus suggests reconstruction wins on hard reasoning and loses on plain retrieval.
This explores when it's cheaper to just dump raw context into the model versus making it actively rebuild what it needs. The honest answer the corpus gives: active reconstruction pays for itself exactly when the work is *reasoning*, and overcharges you when the work is *retrieval*.
The case for reconstruction is strongest when traversal prunes. MRAgent interleaves reasoning with memory traversal and beats retrieve-then-reason by up to 23% — crucially while *lowering* token and runtime cost, because accumulated evidence lets it abandon dead paths instead of stuffing everything into the prompt Can agents reconstruct memory on demand instead of retrieving it?. The same logic shows up in memoryless reasoning: Atom of Thoughts contracts each step so the state depends only on the current sub-problem, shedding the historical baggage that bloats a naive context dump Can reasoning systems forget history without losing coherence?. When reconstruction lets you carry less, it's cheaper.
But reconstruction flips to a tax the moment the task is verbatim copying or simple lookup. Two-layer transformers can copy exponentially long strings essentially for free, while anything that compresses context into a fixed-size state is provably worse at it Can state-space models match transformers at copying and retrieval?. So if you just need the model to *quote the context back*, dumping it is not only cheaper — actively reconstructing into a compressed internal representation actively destroys the thing you wanted. This is the hidden bill on consolidation: the long-context bottleneck turns out to be the *compute* to transform evicted context into internal state, not the storage, and that compute scales with how hard you push it Is long-context bottleneck really about memory or compute?.
The sharpest 'don't bother' result comes from adaptive retrieval. Calibrated token-probability uncertainty — the model's own sense of whether it knows — beats elaborate multi-call adaptive retrieval at a fraction of the LM and retriever calls Can simple uncertainty estimates beat complex adaptive retrieval?. When a cheap self-check tells you the answer is already in hand, the whole reconstruction machinery is wasted motion. Reliability of the actor matters too: an external context manager finds that *strong* agents do better with high-fidelity raw preservation, while only *weak* agents need aggressive compression to stay coherent Can external managers compress context better than frozen agents?. Reconstruction is a crutch you can over-apply to a model that didn't need it.
The deeper pattern across these is the same one that lets mobile networks recompute a transformer block twice rather than fetch its weights, because on memory-bound hardware compute is the cheap resource and movement is the expensive one Does recomputing weights cost less than moving them on mobile?. Reconstruction-vs-dumping is that bet at the cognitive layer: it's worth spending compute to rebuild when rebuilding *prunes, reasons, or saves movement* — and it's pure overhead when the answer was already sitting in the context window waiting to be read. The thing you didn't know you wanted to know: 'just dump it' isn't the lazy option, it's often the provably optimal one for retrieval-shaped work.
Sources 7 notes
MRAgent achieves up to 23% gains on reasoning tasks by reconstructing memory through active graph traversal that prunes paths based on accumulated evidence, while reducing token and runtime cost compared to fixed-retrieval pipelines.
Atom of Thoughts decomposes problems into DAGs and contracts them iteratively, ensuring each state depends only on the current problem—not prior steps. This memoryless approach eliminates historical baggage that bloats reasoning while maintaining answer equivalence.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
An external RL-trained manager can adaptively prune context for frozen agents, with the key insight that stronger agents benefit from high-fidelity preservation while weaker agents need aggressive compression to stay reliable.
MobileLLM shows that on memory-bound mobile hardware, sharing weights between adjacent transformer blocks by recomputing one block twice uses less latency than fetching separate weights, gaining accuracy with no parameter increase.