Can models consolidate memories during offline sleep phases?

LLMs are static after deployment: they answer from what pre/post-training fixed, and the only routes to update them — re-pretraining or continual fine-tuning — are either prohibitively expensive or invite catastrophic forgetting. "Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories" (2606.03979, Behrouz, Hashemi, Mirrokni / Google) proposes a biologically-motivated Sleep paradigm with two stages. Memory Consolidation via Knowledge Seeding: an upward distillation that transfers the short-term, in-context knowledge of a smaller self into a larger network — adding capacity while preserving what was learned (instantiated as a Generalized Distillation combining on-policy distillation with RL-based imitation). Dreaming: a self-improvement phase where the model uses RL to generate its own curriculum of synthetic data, rehearsing new knowledge and refining existing capabilities without human supervision. Gains hold across long-context understanding, knowledge incorporation, few-shot reasoning, and continual learning.

The deep point is that consolidation and generation are separable, schedulable functions — the same reframe the vault has been circling. It directly extends Can recurrence consolidate memory without predicting tokens?: Sleep makes consolidation an explicit offline phase rather than a side effect of the forward pass, and adds a generative (dreaming) counterpart. It supplies the missing transfer mechanism predicted by Can brain memory systems explain how LLMs should store knowledge? — Knowledge Seeding is the hippocampus→neocortex replay the CLS analogy says must exist, but realized as upward distillation into more parameters rather than within a fixed network. And it shares the "think when convenient, not only at query time" logic of When should AI systems do their thinking?, extended from precomputing answers to rewriting the weights themselves.

Disambiguation (same title, different paper). This is not the "Language Models Need Sleep" cited in Is long-context bottleneck really about memory or compute? (arXiv 2605.26099), whose "sleep" is offline recurrence over evicted KV-cache to convert context into internal state. Behrouz et al. (2606.03979) instead consolidate via upward distillation into a larger network plus an RL dreaming curriculum. Two papers, identical title, complementary mechanisms — both treat sleep as the moment compute reorganizes memory, but one solves the long-context eviction bottleneck and the other solves lifelong continual learning.

Relevant Notes

• Can recurrence consolidate memory without predicting tokens? — Sleep makes consolidation an explicit offline phase and adds a generative dreaming counterpart
• Can brain memory systems explain how LLMs should store knowledge? — Knowledge Seeding is the CLS-predicted replay mechanism, realized as upward distillation
• Is long-context bottleneck really about memory or compute? — the OTHER same-titled paper (2605.26099); distinct mechanism, cross-linked for disambiguation
• When should AI systems do their thinking? — same think-when-convenient logic, extended from precomputing answers to rewriting weights