Can neural memory modules scale language models beyond attention limits?
Can separating short-term attention from adaptive long-term memory allow models to efficiently handle context windows exceeding 2M tokens while maintaining competitive performance?
Titans (2501.00663) introduces a neural long-term memory module that addresses a fundamental contradiction in linear recurrent models: they are designed for efficiency on long contexts, but long contexts cannot be properly compressed into small fixed-size states.
The architectural insight is that attention and memory serve fundamentally different functions. Attention operates as short-term memory — accurate direct dependency modeling within the current context window, but quadratic cost limits its reach. Neural memory operates as long-term memory — compressed and persistent, memorizing data that is surprising or close to surprising tokens. The memory update mechanism considers the proportion of memory size to data surprise, resulting in adaptive memory management.
Three integration variants are proposed: memory as context (attending to memory alongside current context), memory as gating (memory modulates attention output), and memory as a layer (memory replaces some attention layers). Each variant trades off between integration depth and computational overhead.
The results establish that Titans outperform both standard Transformers (with the same context window) and modern linear recurrent models across language modeling, common-sense reasoning, genomics, and time series. Critically, Titans scale to context windows larger than 2M tokens while showing competitive performance with Transformers that use the entire context — the long-context problem is addressed without the quadratic penalty. The persistent nature of the memory module makes it a natural substrate for Can models precompute answers before users ask questions? — the memory can store precomputed inferences between interactions, and sleep-time processing can populate the memory with anticipated query-relevant information.
Since Can models reason without generating visible thinking tokens?, the Titans architecture offers a complementary path: rather than scaling reasoning depth through recurrent computation, it scales memory breadth through adaptive memorization. Both bypass the limitations of standard attention but along different architectural dimensions.
Miras unifying framework (2504.13173): The "It's All Connected" paper reconceptualizes Transformers, Titans, and modern linear recurrent models as associative memory modules that learn a mapping of keys to values using an internal objective — termed "attentional bias." The paper observes that most existing sequence models use either dot-product similarity or ℓ2 regression objectives as their attentional bias. Miras provides a general framework with four design choices: (i) associative memory architecture, (ii) attentional bias objective, (iii) retention gate, and (iv) memory learning algorithm. Forgetting mechanisms are reinterpreted as retention regularization — providing a principled basis for forget gates across architectures. Three novel sequence models — Moneta, Yaad, and Memora — go beyond existing linear RNNs while maintaining fast parallelizable training. Different Miras configurations yield models with varying strengths: some excel at language modeling, others at commonsense reasoning or recall-intensive tasks. This generalizes the Titans insight: the attention-as-short-term/memory-as-long-term distinction is one instance of a broader design space where attentional bias objective and retention mechanism can be independently varied.
Inquiring lines that read this note 149
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How does offloading cognition to AI assistance affect learning retention?- How do users perceive attention from systems that lack continuous temporal presence?
- Can better attention mechanisms close the gap between human and AI frame-activation?
- How does AI's inability to sustain temporal attention limit its capacity for expert roles?
- How does the temporal structure of attention differ between humans and AI?
- Does transformer attention architecture fundamentally prevent topic-aware memory?
- Why does attention-based drift happen automatically during generation?
- What does attentional state look like in a static context window?
- How does circuit complexity limit which grammatical structures transformers can acquire?
- How do attention heads separate text retrieval from internal thought representation?
- Can targeted interventions on attention heads bridge the encoding-generation gap?
- Why do transformer attention patterns show positional and sequential bias across tasks?
- How do neural memory modules extend context length beyond attention limits?
- Why does transformer attention architecture undermine stickiness in model behavior?
- How does transformer attention bias toward repeated and context-prominent content?
- Do transformer architectures structurally bias models toward short-term optimization?
- How does attention sink behavior relate to internal model architecture?
- Why does attention concentrate on the first 25% of long input sequences?
- Does attention linearity alone explain the efficiency gains over standard transformers?
- How do attention mechanisms fail at capturing graph structure?
- What computation remains in the attention heads that programs cannot capture?
- How does transformer attention structurally bias models toward prominent and repeated content?
- Can context compression preserve what matters without introducing bias?
- Can long-context readers handle compositional tasks or just semantic search?
- How do retrieved memories differ from decision-context passages for prediction?
- Do attention scores predict which tokens will be pruned first?
- How does completion-driven KV pruning differ from attention-based cache management?
- Why does attention quality degrade as context length increases?
- How do model priors enable targeted context queries without full attention?
- What makes multi-session context tracking harder than single-turn underspecification problems?
- How does context budget create tradeoffs between memory and skills?
- How do memorization and attention map onto different memory systems?
- How does separating local and global context dependencies affect long-context performance?
- What mechanisms cause short contexts to degrade more under aggressive sparsity?
- Which attention heads are essential for maintaining factuality in sparse models?
- Can models consolidate context into weights during idle offline phases?
- Does including full context always degrade memory retrieval quality in practice?
- Does recurrent memory or gist compression work better for ultra-long context?
- How do memory hierarchies and compression reduce context management demands?
- Can recurrent state mechanisms process longer sequences than attention-based working memory approaches?
- How do adaptive memory modules compare to feedback-based working memory for long context?
- Why does attending to own latents work better than bolted-on external memory stores?
- Can adaptive memory modules combine long-term filtering with short-term attention benefits?
- What task profiles favor recurrent filtering over scaled attention mechanisms?
- How does externalized state affect the long-context bottleneck in language models?
- How does reducing activation precision further extend context length?
- How do recurrent memory systems handle ultra-long context differently than attention?
- What are the concrete efficiency gains of linear-attention state-space models?
- Can fixed-size latent states losslessly store arbitrary input context?
- Why does attention excel at context retrieval but struggle with state updates?
- How does accumulated context history degrade iteration quality in long-horizon tasks?
- What capacity limits does the memory model face as corpus grows?
- What makes sparse attention more reliable for long-context retrieval?
- How do memory relevance filters fail to prevent performance degradation?
- Can compressed long-term memory outperform fixed-window token retention?
- How do compressed persistent memory states inside networks compare to attention for long context?
- Why does context compression need reward signals beyond explicit coherence metrics?
- Why does KV cache memory waste grow worse with longer sequences and larger models?
- Do retrieval-augmented memory systems actually solve the compartmentalization problem?
- How should memory consolidation timing differ across multiple timescales?
- Why does recency-based recall outperform semantic similarity for episodic memory?
- What persistent memory architectures best support storing precomputed inferences across sessions?
- What computational costs does closed-loop memory refinement introduce?
- Why does LLM memory consolidation regress below no-memory baselines?
- Can episodic raw memory outperform consolidated summaries in practice?
- Can offline recurrent passes replicate sleep-based memory consolidation in AI?
- How does the hippocampus bind disparate elements without storing everything itself?
- Why is consolidation quality the binding constraint in neural memory systems?
- Why do language models ignore condensed memory even when it is the only memory?
- How should memory systems split between short-term and long-term storage?
- Can continuum memory systems prevent catastrophic forgetting in neural networks?
- How much does memorization capacity limit a model's ability to learn new information?
- How does dual-rate learning separate episodic and procedural memory in neural networks?
- Can data pruning strategies exploit the finite nature of memorization capacity?
- What attentional bias objectives compete with dot product similarity for associative memory?
- How do retention gates regularize forgetting across different sequence model architectures?
- Can the joint-training principle extend beyond memorization and generalization pairs?
- How can memory shift from a passive datastore to an actively trained component?
- Can neural modules memorize surprising tokens as adaptive long-term memory?
- What mechanism transfers explicit memories into parametric model weights?
- Can memory-based adaptation and gradient fine-tuning operate on complementary timescales?
- How do complementary learning systems explain the need for fast and slow consolidation?
- Do long-term memory modules outperform consolidation into fast weights?
- How can a forgetting policy preserve rare knowledge while preventing over-generalization?
- What is the theoretical capacity limit before memorization saturates?
- How do fixed recurrent states trade off copying accuracy for filtering ability?
- When does training a memory model beat RAG or fine-tuning?
- Can native memory procedures acquired through training handle stale or incorrect cached information?
- Does moving memory outside model weights avoid the limitations of in-weight retention?
- Can in-weight memorization scale beyond model parameter count limits?
- What makes multimodal conditioning effective when features are decomposed to the right granularity?
- What neural or architectural mechanism allows selective override of frequency effects?
- What makes sparse models inefficient to train and deploy at scale?
- Why do cross-product features memorize better than dense embeddings?
- Does conditional memory reduce computation alongside conditional sparsity?
- Can memory primitives become first-class design objects like computation sparsity?
- Why do longer sequences tolerate higher sparsity than shorter ones?
- Why do hybrid memory and compute sparsity outperform pure parameter scaling?
- Do KANs maintain their advantages in deep architectures and large-scale training?
- What makes looped latent computation more efficient than scaling attention capacity?
- Can spiking sparsity replace weight quantization as a primary efficiency lever?
- Can attention linearity achieve similar efficiency gains as weight quantization?
- Why do hybrid attention architectures outperform pure linear attention models?
- Does static per-token sparsity repeat the fixed-budget mistake at short sequences?
- Can retrofitted sparse attention ever match jointly-trained sparse attention?
- Can transformer attention patterns actually prevent topic context loss in practice?
- Does attention decay per turn apply equally across different model sizes?
- Can fast-slow separation improve both memory and generation in language models?
- Does the prediction unit shape what language models actually learn?
- Does attention bias explain grounding failure in language models?
- What other internal model decisions beyond attention could be optimized directly?
- Can language models execute iterative numerical methods in latent space?
- How should tiny language models be architected differently than large ones?
- Can externalized memory and skills replace model scaling?
- Does memory granularity need to match the task domain or the model size?
- Can a model be strong at MMLU but weak at long-horizon tasks?
- How should benchmark design account for task-dependent sparsity tolerance differences?
- Why do language models fail at iterative numerical optimization despite scale?
- Do pretrained language models carry reusable computational scaffolding for length handling?
- Which specific capabilities must AI develop beyond current language model abilities?
- Can zero-weight drift through external memory replace parameter plasticity entirely?
- What prevents individual session learning from scaling system-wide?
- Can test-time scaling compound through memory consolidation into a new scaling law?
- Can sleep-time compute reduce latency demands during model inference?
- How do sleep-time and post-completion methods reduce inference latency?
- How does conditional memory compare to conditional computation for efficiency?
- What are the scaling law differences between vision and language learning?
- How do parameter scaling and latent vectors interact in language models?
- What limits the capacity of context-based fast adaptation channels?
- How much does sliding-window augmentation improve single-session modeling?
- Can latent recurrence overcome the trainability costs of depth?
- Can latent recurrence achieve the depth that standard transformers cannot?
Related concepts in this collection 6
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Can models reason without generating visible thinking tokens?
Explores whether intermediate reasoning must be verbalized as text tokens, or if models can think in hidden continuous space. Challenges a foundational assumption about how language models scale their reasoning capabilities.
complementary architectural innovation: depth-recurrent reasoning vs. breadth-adaptive memory
-
Can models reason without generating visible thinking steps?
Do machine reasoning systems actually require verbalized chains of thought, or can they solve complex problems through hidden computation? This challenges how we measure and understand reasoning.
alternative to verbalized reasoning
-
Can models precompute answers before users ask questions?
Most LLM applications maintain persistent state across interactions. Could models use idle time between queries to precompute useful inferences about that context, reducing latency when users actually ask?
Titans' persistent neural memory is a natural substrate for sleep-time compute: the memory module can store precomputed inferences between interactions, and sleep-time processing can update the memory with anticipated query patterns; both exploit statefulness to reduce per-query cost
-
Can latent thought vectors scale language models beyond parameters?
Explores whether explicit latent thought vectors with dual-rate learning create new scaling dimensions independent of model size. This matters because it suggests alternatives to simply building larger models.
LTMs implement fast-slow dynamics at the generation level (dual-rate learning of thought vs token vectors) complementing Titans' fast-slow at the memory level (attention as short-term, neural memory as long-term)
-
Can lookup memory and computation work together better than either alone?
Mixture-of-Experts handles dynamic logic, but static knowledge might need a different mechanism. Can a hybrid approach combining conditional computation with fast lookup outperform pure sparse models?
Engram adds static O(1) N-gram lookup as a third memory primitive complementing Titans' neural memory; together they suggest memory architecture has multiple co-equal axes (neural-adaptive, static-lookup, attention) rather than a single hierarchy
-
Has memory architecture replaced parameter count as the scaling frontier?
Late-2025 research suggests the field's next major efficiency gains come from restructuring how models store and use experience rather than simply making them larger. Three convergent signals point to this shift.
Titans/Miras is one of three convergent signals (alongside ReasoningBank MaTTS and Engram U-curve) that memory-architecture restructuring has displaced parameter count as the active scaling axis
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Titans: Learning to Memorize at Test Time
- Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
- In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss
- Language Models Need Sleep
- Repeat After Me: Transformers are Better than State Space Models at Copying
- Memorization and Knowledge Injection in Gated LLMs
- Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
- Jamba: A Hybrid Transformer-Mamba Language Model
Original note title
neural memory modules that adaptively memorize surprising tokens complement attention as long-term vs short-term memory — scaling to 2M+ context