Do networks recover from forgetting before re-encountering documents?

The default story of sequential training is catastrophic interference: forgetting increases monotonically as a network trains on a sequence of different documents. This paper studies a structured non-IID setting — documents presented cyclically in a fixed, repeated order — and finds a remarkable opposite phenomenon: anticipatory recovery. Networks recover from the forgetting of a document before they encounter it again in the cycle, as if pre-positioning themselves for what's coming. The effect emerges and becomes more robust as the model scales up parameters, and only when each document is well-fitted before moving on; visualizations of weights, activations, and gradients show clear temporal structure.

The keeper is that over-parameterized networks in structured, repeating environments behave unlike the catastrophic-interference picture — they exploit the temporal regularity of the training schedule to organize their weights anticipatorily. This is closer to how humans learn from structured, repeating material than the random-sampling default of LLM pretraining.

This adds a training-dynamics surprise to the vault. It connects to the broader theme that structure in the learning process matters, alongside Does teaching question patterns before document training improve knowledge access? (order of encoding shapes outcomes) and Is LLM forgetting really knowledge loss or alignment loss? (forgetting is often recoverable, not destruction) — both complicate the simple catastrophic-forgetting narrative.