Can recurrent memory scale where attention fails on ultra-long text?

BABILong is a leak-proof benchmark for extracting and processing facts distributed across very long texts (length and placement are algorithmically adjustable, so future LLMs can't have memorized it). Two findings stand out. First, common methods — including GPT-4 and RAG — are effective only for sequences up to ~10⁴ elements, and their performance relies heavily on the first 25% of the input: a stark quantification of the lost-in-the-middle problem, where attention effectively ignores the bulk of a long context. Second, fine-tuning a small GPT-2 with recurrent memory augmentation lets it handle up to 11 million tokens — by far the longest input processed by any neural model — and crucially enables multi-hop reasoning by filtering irrelevant information rather than attending over everything.

The keeper is the comparative claim: recurrent memory excels at filtering irrelevant content in a way that scaling attention does not. Where attention degrades and concentrates on the start of the input, a compact recurrent state forces the model to decide what to carry forward — and that selectivity is what unlocks ultra-long multi-hop reasoning.

This complements the vault's long-context thread from the memory side. It pairs with Can neural memory modules scale language models beyond attention limits? (Titans) as another recurrent-memory route past attention's limits, and it is the empirical lost-in-the-middle ground for How do LLMs balance remembering context versus keeping it separate?. It also sits in productive tension with Can state-space models match transformers at copying and retrieval?: a fixed recurrent state is worse at verbatim copying yet better at filtering for multi-hop fact extraction — the task profile decides which wins.