Can recurrent state mechanisms process longer sequences than attention-based working memory approaches?
This explores whether recurrent mechanisms — models that carry a compressed running state forward — can handle longer inputs than attention, which compares every token to every other, and what each approach trades away.
This explores whether recurrent mechanisms — models that carry a compressed running state forward — can handle longer inputs than attention, which compares every token to every other, and what each approach trades away. The corpus says yes, on raw length, recurrence wins decisively: a fine-tuned GPT-2 with recurrent memory augmentation processes up to 11 million tokens by *filtering* irrelevant content rather than attending to all of it, exactly where attention-based models degrade and fixate on the start of the input Can recurrent memory scale where attention fails on ultra-long text?. The reason attention struggles isn't mysterious — its cost grows with the square of the sequence, so doubling the input quadruples the work, which is why architectures that scale to millions of tokens lean on a compressed state instead.
But length isn't the whole story, and here the corpus pushes back in a way you might not expect. There's a provable limit baked into recurrence: because a recurrent model squeezes everything into a *fixed-size* state, it can't faithfully copy or retrieve long spans the way attention can. Two-layer transformers can copy exponentially long strings; state-space models hit a wall set by how much their latent state can hold Can state-space models match transformers at copying and retrieval?. So the honest answer is a split decision — recurrence reads further, attention remembers verbatim better. Length and fidelity are different axes.
The most interesting moves in the corpus refuse the either/or. Titans bolts a neural long-term memory onto attention, keeping attention for short-range work while a separate module adaptively memorizes *surprising* tokens, scaling past 2M context without the quadratic penalty Can neural memory modules scale language models beyond attention limits?. TransformerFAM goes further with no new weights at all — it loops the transformer's attention back onto its own latent representations, growing an emergent working memory for indefinitely long inputs Can models learn working memory by attending to their own latents?. And ReadAgent sidesteps the architecture question entirely by acting human: it compresses documents into 'gist memories' up front and fetches details only when a task demands them, stretching effective context 3–20× Can LLMs read long documents like humans do?.
Here's the thing you didn't know you wanted to know: a recent line of work argues the real bottleneck was never memory *capacity* at all — it's the *compute* needed to fold evicted context into the model's internal state Is long-context bottleneck really about memory or compute?. Recurrence may be valuable less for storage and more as a *consolidation* tool — running extra passes with no new input to transfer recent context into persistent fast weights, mirroring how the hippocampus replays memories during sleep Can recurrence consolidate memory without predicting tokens?. That reframes the whole question: it's not 'which mechanism holds more,' but 'which mechanism spends compute to turn experience into durable state.'
One sobering footnote keeps both camps humble. Reasoning accuracy collapses long before any architecture runs out of room — dropping from 92% to 68% with just 3,000 tokens of padding, far under the context limit Does reasoning ability actually degrade with longer inputs?. Processing a longer sequence and *reasoning well over* a longer sequence are not the same achievement, and so far no mechanism here fully closes that gap.
Sources 8 notes
Fine-tuned GPT-2 with recurrent memory augmentation processes up to 11 million tokens and enables multi-hop reasoning by selectively filtering irrelevant content, where attention-based models degrade and concentrate on early input.
Two-layer transformers can copy exponentially long strings while state-space models are fundamentally limited by their fixed-size latent state. Empirically, transformers dramatically outperform SSMs at copying and context retrieval in both synthetic and pretrained settings.
Titans architecture separates attention (short-term, quadratic) from neural memory (long-term, compressed), prioritizing surprising tokens for storage. The model outperforms standard Transformers and linear RNNs across tasks while scaling to 2M+ token contexts without quadratic penalties.
TransformerFAM demonstrates that adding a feedback loop lets transformers attend to their own latent representations, fostering emergent working memory for indefinitely long inputs. The approach requires no additional weights and improves long-context performance at 1B, 8B, and 24B scales.
ReadAgent compresses documents into gist memories before knowing the task, then retrieves details only when needed, extending effective context 3–20× and outperforming retrieval baselines on long-document QA.
Research shows the bottleneck is not memory capacity but the compute required to consolidate evicted context into fast weights during offline sleep phases. Performance improves with more consolidation passes, following a test-time scaling pattern on harder reasoning tasks.
Language models can use recurrent passes without input tokens to transfer recent context into persistent fast weights via learned local rules, mirroring hippocampal replay during biological sleep. This separates consolidation from prediction, enabling different scheduling and compute allocation.
FLenQA shows reasoning accuracy drops from 92% to 68% at just 3000 tokens of padding, far below context window capacity. The degradation is task-agnostic, uncorrelated with language modeling performance, and persists even with chain-of-thought prompting.