Can adaptive memory modules combine long-term filtering with short-term attention benefits?
This explores whether a model can split memory into two cooperating channels — a fast, attention-based short-term store and a selective long-term memory that filters what's worth keeping — rather than forcing one mechanism to do both jobs.
This explores whether a model can split memory into two cooperating channels — fast attention for the recent stuff, and a slower long-term store that filters what's worth keeping — and whether the combination actually works. The corpus's clearest 'yes' is the Titans architecture Can neural memory modules scale language models beyond attention limits?, which does exactly this: it keeps standard attention for short-term, high-resolution recall (but pays the usual quadratic cost, so it stays small) and bolts on a neural memory module that compresses the long horizon. The filtering trick is what makes it adaptive — instead of storing everything, it prioritizes *surprising* tokens, the ones that violate prediction and therefore carry the most information. That single design choice lets it stretch past two million tokens of context without the cost blowing up, and beat both plain Transformers and linear RNNs.
The surprise-based filter isn't an isolated idea — the corpus suggests models already do something like adaptive filtering internally. When tasks get unfamiliar, hidden states sparsify in a systematic way that behaves like a selective filter, stabilizing performance under distribution shift rather than breaking down Do language models sparsify their activations under difficult tasks?. And a tiny number of 'massive activations' quietly act as implicit attention bias, steering where attention concentrates Do hidden massive activations act as attention bias terms?. So the two-channel design is partly formalizing filtering instincts the architecture already has.
There's a cheaper way to get the short-term benefit without a separate memory bank: let the model attend to its *own* latent representations through a feedback loop. TransformerFAM does this and grows an emergent working memory for arbitrarily long inputs — with no extra weights at all Can models learn working memory by attending to their own latents?. That's a useful contrast to Titans: one adds a dedicated long-term module, the other recycles the network's own activations as a rolling scratchpad. Both are betting that short-term attention and long-term retention are genuinely different jobs that shouldn't share one mechanism — the same bet shows up in continual-learning work that routes fast lessons into prompts and slow ones into weights to avoid forgetting Can splitting adaptation into two channels reduce forgetting?, and in the 'sleep phase' idea where in-context knowledge gets consolidated into weights offline so it persists without overwriting what's already there Can models consolidate memories during offline sleep phases?.
But here's the thing the question doesn't anticipate: combining channels can backfire when the long-term store keeps *reprocessing* itself. COMEDY folds memory generation, compression, and response into one model and drops retrieval entirely — elegant in principle — yet empirically it follows an inverted-U curve, eventually degrading *below* a no-memory baseline because continuous re-compression causes misgrouping, context loss, and overfitting Can a single model replace retrieval for long-term conversation memory?. The counter-lesson comes from Reflexion, where keeping memories *uncompressed* — storing verbal reflections verbatim in episodic memory — is what preserves their usability Can agents learn from failure without updating their weights?.
So the answer is yes, adaptive memory modules can combine the two — but the filter is the whole game. Titans wins because surprise-prioritization decides *what* to keep before compression happens; COMEDY stumbles because it compresses indiscriminately and repeatedly. The unexpected takeaway: the benefit of a long-term channel isn't storage capacity, it's a good forgetting policy. A memory that can't decide what to throw away is worse than no memory at all.
Sources 8 notes
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
A very small number of input-agnostic activations with values up to 100,000× larger than others act as indispensable implicit bias terms and concentrate attention probability onto specific tokens. This phenomenon appears across model sizes and Vision Transformers.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
Fast-Slow Training routes task-specific lessons into optimized prompts while keeping parameter updates minimal, reaching equivalent performance 1.4–3x faster with substantially less catastrophic forgetting and plasticity loss, demonstrating that forgetting is a misallocation problem rather than an inherent cost.
The Sleep paradigm uses Knowledge Seeding (distilling smaller networks into larger ones) and Dreaming (RL-generated rehearsal) to consolidate in-context knowledge into weights without forgetting. Gains appear in long-context understanding, few-shot reasoning, and continual learning.
COMEDY merges memory generation, compression, and response into one operation, tracking event recaps, user portraits, and relationship dynamics without vector-DB retrieval. However, empirical work shows continuous reprocessing follows an inverted-U curve, degrading below no-memory baseline due to misgrouping, context loss, and overfitting.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.