Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations

On-policy distillation (OPD) has become a key paradigm in LLM post-training, yet its training dynamics remain poorly understood. We present a systematic study examining the role, pathologies, and regulations of OPD. We first clarify the role of OPD as an exploration catalyst: it steers the student toward correct reasoning paths via dense token-level guidance, without expanding capability ceiling. We confirm this by showing that prompt diversity matters more than per-problem sampling numbers, and critically, that the effectiveness of OPD hinges entirely on the quality of its guiding signal. This dependency exposes two pathologies that derail exploration. The Student-Teacher Mismatch occurs when a large teacher-student distributional gap causes the guiding signal to misalign with task correctness, steering exploration in counterproductive directions. Length Exploitation arises when the aggregated tokenlevel objective creates length-dependent shortcuts, allowing the student to game the reward landscape through response truncation or redundant padding, exploring degenerate length modes rather than reasoning strategies.

Introduction. On-policy distillation (OPD) has rapidly become a standard component of modern LLM post-training pipelines. By providing token-level supervision from a stronger teacher over the student’s own rollouts, OPD has demonstrated remarkable effective- ness (Gu et al., 2026; Yang et al., 2025; DeepSeek- AI, 2026). Despite its growing adoption, the fundamental training dynamics of OPD remain poorly understood. Practitioners often observe highly inconsistent behaviors: OPD sometimes improves performance (Lu and Lab, 2025), yet in other cases becomes unstable (Luo et al., 2026; Zhu et al., 2026), collapses exploration, even underperforms outcome-based reinforcement learning (Li et al., 2026a). This raises our systematic empirical study, we aim to (1) explore what benefits OPD brings (Role§ 3), (2) when OPD fails (Pathologies§ 4), and (3) how to improve it from the previous findings (Regulations§ 5).

Discussion / Conclusion. In this work, we look inside OPD to shift the paradigm from empirical scaling to rigorous signallevel diagnosis. We first established that OPD acts as an exploration catalyst which benefits more from problem diversity. Then we systematically expose two critical pathologies: length exploitation via redundant token padding, and the Student- Teacher Mismatch, where severe capabilities mismatch causes negative impacts. To restore guidance fidelity, we introduce lightweight signal regulations that require zero off-policy overhead. Our empirical results demonstrate that regulating token-level advantage dynamics effectively eliminates these failures, enabling a 4B teacher to surpass configurations distilled from larger models, which is signal quality, not teacher scale, governs OPD success.