Can an AI squeeze a huge history into a small running memory, or does it still need to reread everything?

This explores whether squeezing a long history into a compact, persistent state inside the network can stand in for attention, which keeps every token in view. The corpus suggests they aren't really rivals. Compression buys cheap length and gives up exact recall, and the strongest designs keep attention for precision and add a compressed channel for reach.

Attention wins on exactness, and there's a proof. Two-layer transformers can copy exponentially long strings, while state-space models are capped by their fixed-size latent state, and transformers dramatically outperform them on copying and context retrieval in practice too Can state-space models match transformers at copying and retrieval?. Any fixed-size state is a lossy summary by construction. If your task needs the exact phone number from page 300, a compressed state has to have guessed in advance that you'd want it.

That is why the interesting architectures mix the two. Titans gives attention the short-term job and hands long-term storage to a neural memory that prioritizes surprising tokens. It beats standard Transformers and linear RNNs and scales past 2M tokens without the quadratic penalty Can neural memory modules scale language models beyond attention limits?. TransformerFAM takes a different route: it feeds the model's own latents back so attention can attend to them, and working memory emerges for indefinitely long inputs with no extra weights Can models learn working memory by attending to their own latents?. In both cases the compressed state is something attention reads, not a replacement for it. Metis pushes the same idea further, arguing the memory state and its update procedures belong inside the backbone and should be trained end-to-end. External memory modules and the backbone otherwise optimize separately and drift apart Should agent memory live inside the model backbone?.

The less obvious finding is where the cost goes. One line of work argues the long-context bottleneck isn't memory capacity but the compute needed to turn evicted context into internal state. Consolidating it into fast weights during offline sleep phases improves with more passes, following a test-time-scaling pattern on harder reasoning tasks Is long-context bottleneck really about memory or compute?. So compression isn't free. It moves the expense from every query (attention) to a one-time digestion step. Agent-level systems make the same trade in plain language. ReadAgent compresses documents into gist memories before it knows the task, then looks up details only when needed, which extends effective context 3–20x Can LLMs read long documents like humans do?. DeepAgent folds its own history into episodic, working, and tool schemas, and the autonomy and structure are what keep the compression from degrading it Can agents compress their own memory without losing critical details?. In both, a lossy gist decides where to look and exact retrieval supplies the detail.

There's a warning about the read side. MemTrapBench found that all five tested memory frameworks underperformed a no-memory baseline by more than 10%, even though the memories were accurately stored and relevant Can relevant memories actually harm LLM reasoning?. Good compression doesn't help if the model misuses what it recalls. The corpus doesn't have a head-to-head test of consolidation-style fast weights against Titans-style memory modules on the same tasks, so the strongest direct comparison is the theoretical one: fixed-size state loses to attention on exact retrieval, and the hybrids exist because of that.