An AI teacher can only correct a student it's close to — drift too far and the feedback signal collapses.
What makes student-teacher distributional mismatch derail on-policy distillation?
This explores why the gap between a teacher's output distribution and a student's own generating distribution is the thing that breaks on-policy distillation — and what specifically goes wrong when the two drift too far apart.
This is about a specific failure: on-policy distillation only works because the student learns on sequences *it* generated, with the teacher scoring them — so when the teacher's distribution sits too far from the student's, the correction signal stops landing. The whole point of on-policy methods like GKD is to fix the classic train/inference mismatch by aligning training data with the student's actual behavior at inference time Can distillation work on the student's own generated sequences?. That alignment is fragile in exactly one direction: the teacher has to stay reachable from where the student already is. When it doesn't, the gradients either fight the student or wash out.
The cleanest account of *why* mismatch derails things is variance. A distant teacher produces high-variance gradients — the student is being pulled toward regions it rarely visits, so updates are noisy and improvement is non-monotonic. The fix is to stop distilling from a far target and instead construct a *proximal* teacher bounded by a trust region, which caps how far each step can pull and turns unstable training into monotonic improvement at zero extra compute Can proximity between teacher and student fix distillation instability?. This reframes the problem: distillation instability isn't about teacher weakness, it's about teacher *distance*.
That also explains a counterintuitive result — bigger, 'better' teachers can make things worse. On-policy distillation is really an exploration catalyst, not a capability expander: it steers the student toward correct paths inside its existing envelope, so signal quality and diversity matter far more than teacher scale Does on-policy distillation actually expand student capability?. A refinement that is objectively higher quality but sits past the student's learning frontier degrades performance, which is why students should filter teacher data against their own statistical profile and keep only what's compatible Does teacher-refined data always improve student model performance?. Mismatch isn't only geometric distance in output space — it's difficulty the student can't yet absorb.
There's a subtler, second-order derailment too: the mismatch can be invisible in-domain and only bite out-of-distribution. Teachers conditioned on the right answer and verifier output emit confident, concise traces with the uncertainty stripped out; the student faithfully inherits that style and looks great on in-domain problems while quietly losing the epistemic caution it needs elsewhere Does richer teacher context hurt student generalization?. So the student matched the teacher's *distribution* and still generalizes worse — the mismatch got transferred rather than closed. A related collapse mechanism shows up when the training signal itself flattens: when within-prompt reward variance is low, task gradients weaken and regularization takes over, pushing outputs toward generic input-agnostic templates Why do language models collapse into generic templates?.
The reader-worth-it twist: mismatch isn't always fatal, and understanding why tells you when to *want* it. Walmart's small cross-encoders actually beat their LLM teachers — because the student saw a far broader input distribution than the teacher ever did, and the teacher's labels merely smoothed it Can smaller models outperform their LLM teachers with enough data?. So the same distributional gap that derails on-policy distillation (student pulled toward an unreachable teacher) can *help* off-policy distillation (student's own wide distribution, teacher used only as a soft label). The derailment is specific to on-policy setups where the student is chasing a moving target it can't reach — bound the distance, match the difficulty, and the mismatch stops being a liability.
Sources 7 notes
Generalized Knowledge Distillation (GKD) fixes the train-inference mismatch by having students generate sequences that the teacher then scores, aligning the training distribution with the student's actual inference behavior. This on-policy approach also composes cleanly with RL fine-tuning.
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Show all 7 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Trust Region Policy Distillation
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Self-distillation Enables Continual Learning
- Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations