Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement

On-policy distillation (OPD) offers dense token-level supervision as an alternative to the sparse outcome-level advantages of reinforcement learning with verifiable rewards (RLVR). However, the teacher scores student-generated trajectories that are inherently off-policy for it, so the reliability of its supervision, and hence the source of the student’s improvement, remains unclear. We quantitatively analyze teacher supervision during OPD training and find substantial noise whose prevalence increases with teacher scale. Surprisingly, the student policy is insensitive to such noise, converging to comparable performance regardless of whether noisy supervision is retained or removed. Does OPD distill at all? By analyzing what drives its gains, we find that learning concentrates on low log-probability tokens, and using a single fixed negative advantage matches the performance of teacherprovided ones. This suggests that OPD works largely by suppressing low logprobability tokens, which requires no teacher. These findings motivate On-Policy Self-Adaptation (OPSA), a supervision-free method using entropy-adaptive negative advantages. It assigns stronger learning signals to high-entropy positions, suppressing tail tokens, and evenly redistributing probability mass among head tokens.

Introduction. Reinforcement learning (RL) has become a dominant post-training paradigm to improve reasoning capabilities of large language models (LLMs) (Guo et al., 2025; Yang et al., 2025; GLM-5-Team et al., 2026), offering stronger performance and generalization than offline methods such as supervised fine-tuning (SFT) (Chu et al., 2025; Chen et al., 2025). Reinforcement learning with verifiable rewards (RLVR), including GRPO (Shao et al., 2024) and DAPO (Yu et al., 2026), samples multiple responses per question and assigns advantages by normalizing verifiable rewards within each group (Fig. 1(a)). However, response-level rewards provide coarse and sparse supervision for long-horizon reasoning (Yue et al., 2025). Moreover, when responses within a group share the same correctness outcome, their normalized advantages vanish (Wang et al., 2026b; Ding et al., 2026), weakening the learning signal and making training unstable or prone to collapse.

Discussion / Conclusion. The central contribution of this work is a systematic investigation into the source of performance gains in OPD. We find that much of OPD’s improvement can be reproduced without teacher supervision by assigning fixed negative advantages to low-probability tokens, while scaling these advantages according to token-level entropy further enhances performance. Building on these findings, we introduce On-Policy Self-Adaptation (OPSA), an external-supervision-free framework that enables policy self-improvement through entropy-adaptive negative advantages. Our analysis shows that OPSA suppresses low-probability tail tokens while redistributing probability mass among competing head tokens. This adaptively sharpens the overall token distribution, improving prediction confidence at low-entropy positions while preserving exploration at high-entropy reasoning forks. As a result, OPSA encourages longer and more reflective reasoning trajectories, leading to substantial improvements in both Avg@32 and Pass@32.