Why does copying a smarter AI become more chaotic at scale, even when the feedback signal is rich and constant?
What causes on-policy distillation to become unstable at scale despite dense rewards?
This explores why on-policy distillation—where a student learns from teacher signal on its own generated rollouts—can destabilize as models and training scale up, even when the reward signal is dense rather than sparse.
This reads the question as being about a specific failure: dense per-token teacher signal should, in theory, give the student rich guidance everywhere, yet training still diverges or stalls at scale. The corpus locates the culprit not in reward density but in the *distance* between teacher and student. When you distill from a far-off, more capable target, the student's on-policy samples land in regions where the teacher's guidance is large and uncorrelated with where the student actually is—so the gradients have high variance and each update yanks the policy around unpredictably. The clearest treatment is TOP-D Can proximity between teacher and student fix distillation instability?, which shows that constructing a *close* teacher bounded by a trust region controls exactly this gradient variance and converts erratic training into monotonic improvement—at no extra compute. Instability, in other words, is a proximity problem, not a signal-quantity problem.
That reframing is reinforced by work arguing on-policy distillation isn't even doing what people assume. It acts as an *exploration catalyst*—steering the student toward correct paths inside its existing capability envelope—rather than lifting the ceiling Does on-policy distillation actually expand student capability?. Once you see it that way, a distant teacher is actively counterproductive: it points toward behaviors the student can't yet reach, so the dense signal becomes noise the student can't act on. Signal *fidelity and diversity* matter more than teacher scale, which is why a smaller, closer teacher often beats a larger one.
The corpus also shows dense rewards degrading in ways that mimic distillation instability, which is worth knowing because the mechanisms rhyme. When within-prompt reward variance is low, the task gradient weakens and regularization takes over, collapsing the policy toward generic, input-agnostic templates Why do language models collapse into generic templates?. Relatedly, RL for reasoning has a predictable entropy-collapse ceiling: as policy entropy falls toward zero, performance saturates and exploratory capacity dies Does policy entropy collapse limit reasoning performance in RL?. Density doesn't protect you from either—a dense but low-variance or entropy-starved signal still drives the model toward a narrow attractor, which at scale reads as brittleness.
There's a subtler failure mode too: what the dense signal *teaches* can be quietly harmful. Teachers conditioned on the correct answer and verifier output produce confident, concise traces that the student inherits, suppressing uncertainty and wrecking out-of-distribution robustness even as in-domain numbers look great Does richer teacher context hurt student generalization?. So a 'strong' dense teacher can bake in overconfidence that only surfaces as instability once you push past the training distribution.
The throughline the reader might not expect: the fixes that work all narrow or filter the signal rather than enrich it—a trust region to bound teacher distance, variance-aware prompt selection to keep gradients informative, entropy management to preserve exploration. The lesson across these notes is that 'dense' is not the same as 'stable.' Stability comes from keeping the teacher reachable and the signal high-variance where it counts, and the same logic shows up wherever people try to convert rich feedback into learnable gradients Can environment feedback replace scalar rewards in policy learning?.
Sources 6 notes
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Show all 6 sources
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Self-distillation Enables Continual Learning
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Trust Region Policy Distillation
- RAGEN-2: Reasoning Collapse in Agentic RL
- Reinforcement Learning via Self-Distillation
- Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
- Efficient Reinforcement Learning via Large Language Model-based Search