Can an AI that boils its past down to a smart summary beat one that just remembers its last few pages word for word?

This explores whether squeezing the past into a compact, learned memory can beat simply keeping the most recent tokens verbatim. The corpus says yes, sometimes, and the deciding factor is how the compression is done. The clearest win is Titans, which splits the job in two. Attention handles the short-term window, and a separate neural memory handles the long-term past. That memory stores tokens in proportion to how surprising they are, so predictable text gets skipped. The result beats standard Transformers and linear RNNs and scales past 2 million tokens without the quadratic cost of attention (Can neural memory modules scale language models beyond attention limits?).

The cost of compression is verbatim recall. Transformers provably beat state-space models at copying and retrieving exact strings, because a fixed-size hidden state can't hold an arbitrarily long input (Can state-space models match transformers at copying and retrieval?). So compressed memory wins where the task needs gist, such as themes, preferences and what happened before. It loses where the task needs the exact sentence from 50,000 tokens ago. Long context has its own limit. It can stand in for retrieval on semantic lookups, but it fails on structured queries that need joins across tables, and more window doesn't fix that (Can long-context LLMs replace retrieval-augmented generation systems?).

One reframing may change how you see the question. One line of work argues the bottleneck isn't storage capacity but the compute needed to turn evicted context into something the model can use. In that work, offline consolidation into fast weights improves with more passes, the same scaling pattern seen in test-time reasoning (Is long-context bottleneck really about memory or compute?). A related Sleep paradigm distills in-context knowledge into weights during offline phases, using rehearsal the model generates itself, and gains on long-context understanding without forgetting (Can models consolidate memories during offline sleep phases?). On this view, compressed memory beats a fixed window only when you spend real effort building it.

The corpus also shows how compression fails. COMEDY folds memory generation, compression and response into one model and drops the retrieval database. But repeated reprocessing follows an inverted-U curve. It helps at first, then drifts through misgrouping and lost context until it scores below having no memory at all (Can a single model replace retrieval for long-term conversation memory?). MemTrapBench is harsher. All five memory frameworks tested fell more than 10% below a no-memory baseline, even though the memories were accurate and relevant (Can relevant memories actually harm LLM reasoning?). A bad memory can be worse than a short window.

Two designs seem to avoid this. In DeepAgent, the agent decides when to fold its history into separate episodic, working and tool memories. That cuts tokens and lets it pause to rethink its strategy, and the corpus credits the autonomy and the structure together for avoiding the degradation (Can agents compress their own memory without losing critical details?). MRAgent skips upfront compression and rebuilds memory on demand by walking a graph and pruning paths as evidence accumulates, gaining up to 23% on reasoning tasks at lower cost (Can agents reconstruct memory on demand instead of retrieving it?). Cost also shifts the comparison. In a 115-day agent deployment, 82.9% of tokens were cache reads, so keeping context around is cheaper than it looks (Do persistent agents really cost less per token?). The corpus has no direct head-to-head between compressed memory and a fixed window across the same tasks, and Titans is the only clean comparison.