Why does preference optimization eventually suppress the preferred outputs?

"Learning Dynamics of LLM Finetuning" decomposes finetuning step-by-step to track how learning one example shifts the model's predictions on other responses. Applied to off-policy DPO, this exposes a counter-intuitive failure the authors call the squeezing effect: when DPO applies a large negative gradient to an already-unlikely rejected output, the probability mass it removes does not flow cleanly to the preferred output — it redistributes across the whole vocabulary in a way that can lower the desired response's likelihood. Therefore running DPO too long makes even the outputs you were trying to promote less probable, which reframes a previously puzzling empirical observation as a mechanical consequence of how negative gradients act on low-probability targets.

The same framework yields a hypothesis for finetuning-induced hallucination: because learning example B shifts predictions on example A, the model can start reusing phrases or facts from B's response to answer A, or lock into repetitive simple phrasings — cross-example contamination as a learning-dynamics artifact rather than a data problem.

This is the mechanistic under-layer for the vault's finetuning-degradation thread. It provides the why behind Does fine-tuning on new facts increase hallucination risk? — negative-gradient dynamics are one channel by which new supervision corrupts old knowledge. It also complements Does every correct chain-of-thought trace improve fine-tuning?: both show that which tokens receive gradient, and for how long, matters more than aggregate loss. The practical takeaway — on-policy variants avoid the squeeze because they don't dump big negatives on stale low-probability samples.