Why does fine-tuning for continuous space cause catastrophic forgetting?
This explores why fine-tuning a model to reason in a continuous latent space (instead of discrete tokens) tends to erase pretrained knowledge — and what the corpus says about the deeper mechanism and the ways around it.
This explores why fine-tuning a model to reason in continuous space (latent 'soft thoughts' rather than discrete words) tends to wipe out what the model already knew. The corpus suggests the problem isn't really about continuous space at all — it's about *where* you put the new learning. When you update the model's own weights to support a new reasoning mode, you overwrite the very parameters that store pretrained knowledge. Several notes converge on the same diagnosis: forgetting is a misallocation problem, not an inherent cost of adaptation.
The sharpest evidence for that reframing comes from work showing that adaptation can be *split into two channels*. Fast-Slow Training routes task-specific lessons into optimized prompts (fast, textual context) while barely touching the weights (slow), and it reaches the same performance 1.4–3x faster with far less forgetting Can splitting adaptation into two channels reduce forgetting?. The mechanism behind why this works is measured directly elsewhere: models that drift less from their base distribution — staying up to 70% closer in KL terms — keep their *plasticity*, the ability to learn the next task at all. Weight-only approaches stall when the domain shifts because they've drifted too far Does staying close to the base model preserve learning ability?. So heavy weight updates don't just erase the past, they degrade the future.
The most direct answer to the continuous-reasoning version of the question is architectural: don't update the backbone at all. SoftCoT *freezes* the main LLM and delegates continuous-thought generation to a small auxiliary model, so the soft reasoning capability is added alongside pretrained knowledge rather than carved out of it Can continuous reasoning avoid forgetting in instruction-tuned models?. The same freeze-the-knowledge instinct shows up in proxy-tuning, which works at decoding time and actually *beats* direct fine-tuning on knowledge tasks — and the paper pinpoints why: direct fine-tuning corrupts knowledge storage in the lower layers, while leaving weights untouched confines change to reasoning and style Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Lower layers are where facts live; continuous-space fine-tuning that reaches them is exactly what hurts.
Beyond freezing, the corpus offers two other escape routes. One is *parameter isolation* — identifying the core regions each task depends on and freezing those while merging the rest, which consistently beats naive multi-task fine-tuning Can isolating task-specific parameters prevent multi-task fine-tuning interference?. The other abandons weight updates entirely: store new skills or experiences *outside* the model. VOYAGER keeps an embedding-indexed library of executable skills and composes new ones from old, learning continuously with no forgetting Can agents learn new skills without forgetting old ones?, while AgentFly pushes policy improvement entirely into episodic memory modules, never touching LLM parameters Can agents learn continuously from experience without updating weights?.
The thread worth taking away: fine-tuning into continuous space causes forgetting because it relocates a *behavior* (how to reason) by overwriting a *store* (what is known) — they happen to share the same weights. There's even a subtler cost lurking here — fine-tuning can make reasoning chains *look* right while no longer actually driving the answer Does fine-tuning disconnect reasoning steps from final answers?. Every successful fix in the corpus works by separating the two: a second model, a prompt channel, an external library, or an untouched base — so the new capability and the old knowledge stop competing for the same parameters.
Sources 8 notes
Fast-Slow Training routes task-specific lessons into optimized prompts while keeping parameter updates minimal, reaching equivalent performance 1.4–3x faster with substantially less catastrophic forgetting and plasticity loss, demonstrating that forgetting is a misallocation problem rather than an inherent cost.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
SoftCoT avoids catastrophic forgetting by keeping the main LLM frozen while delegating soft thought generation to a small auxiliary model. This architectural separation maintains pre-trained knowledge while enabling continuous reasoning.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Research shows that identifying core parameter regions per task, clustering overlapping tasks, and freezing core parameters while geometrically merging non-core parameters consistently outperforms standard multi-task fine-tuning. Temporal task scheduling alone proves insufficient without explicit structural parameter isolation.
VOYAGER demonstrates that storing executable skills in an embedding-indexed library and composing complex skills from simpler ones allows agents to learn continuously while avoiding the forgetting that occurs with weight-update-based methods. Environmental feedback refines skills while an automatic curriculum drives continual exploration.
AgentFly formalizes agent learning as a Memory-augmented MDP with three memory modules (case, subtask, tool) that enable credit assignment and policy improvement entirely through memory operations. The approach achieved 87.88% on GAIA validation without modifying LLM parameters.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.