When an AI keeps learning new jobs and forgets old ones, is that one flaw or several different ones?

This explores whether catastrophic forgetting in sequential fine-tuning is one problem or several. The corpus has no tidy taxonomy, but it says clearly that it's several. One study fine-tuned models on 100 tasks in a row and found that no single mechanism kept strong retention. The authors' hypothesis is that different mechanisms target different sources of forgetting, so combining complementary ones should beat any one fix Can any single continual learning method prevent forgetting?. The sources show up in what each successful workaround is aimed at.

The first source is tasks fighting over the same weights. Identifying each task's core parameter region, freezing it, and merging the rest beats standard multi-task fine-tuning. Just reordering when tasks are trained wasn't enough Can isolating task-specific parameters prevent multi-task fine-tuning interference?. So this interference is built into the structure, and scheduling won't fix it. The second source is damage to what the model already knew before fine-tuning began. Proxy-tuning leaves base weights untouched and shifts outputs at decoding time. It beats direct fine-tuning on knowledge tasks, because direct fine-tuning corrupts knowledge stored in the lower layers, while proxy-tuning mostly affects reasoning and style Can decoding-time tuning preserve knowledge better than weight fine-tuning?. That is a different failure from tasks clashing with each other: it erodes the model's stored facts.

The third source is writing lessons into weights that never needed to go there. Fast-Slow Training sends task-specific lessons into optimized prompts and keeps weight updates small. It gets less forgetting and less loss of plasticity, and its authors call forgetting a misallocation problem, not an inherent cost Can splitting adaptation into two channels reduce forgetting?. The same logic sits behind freezing the backbone and training a small helper to generate soft thoughts Can continuous reasoning avoid forgetting in instruction-tuned models?. It also sits behind VOYAGER's external skill library, which avoids the forgetting that comes with weight-update methods Can agents learn new skills without forgetting old ones?. These approaches sidestep the problem more than they diagnose it. Still, they all work by keeping new learning out of the weights.

Two failures look like forgetting and may not show up in accuracy. Fine-tuned models can hold injected facts yet be unable to use them, because the facts are trapped in early or late layers and never reach the mid-layer reasoning circuits. Relocating those representations recovers 58–75% of the lost generalization Why do fine-tuned models fail to use facts they memorize?. Fine-tuning can also weaken the causal link between a model's reasoning steps and its final answers, so the reasoning becomes performative even when accuracy holds Does fine-tuning disconnect reasoning steps from final answers?. Training order and scale also matter. Models fine-tuned on cyclically repeated documents recover performance on a document before seeing it again, and this grows with model size, so forgetting isn't always a one-way slide Do networks recover from forgetting before re-encountering documents?.

The corpus therefore supports at least these distinct suspects: clashes between tasks over shared weights, erosion of pretrained knowledge, needless weight-writing, hidden routing and reasoning failures, and effects of training order and scale. It doesn't measure how much each one contributes, and it never tests them side by side. That comparison is the open gap the 100-task study points toward.