Trust Region Policy Distillation

Big goals are hard to achieve all at once; breaking them into small steps is wiser. We present Trust Region Policy Distillation (TOP-D), which transforms the notoriously unstable, high-variance On-Policy Distillation (OPD) into a stable training paradigm by dynamically constructing a proximal teacher. Theoretically, we establish a rigorous framework demonstrating that TOP-D inherently controls gradient variance. By providing a formal global convergence analysis alongside a monotonic improvement bound, we mathematically formalize the reliability and stability of the overall training dynamics. Empirically, TOP-D dramatically enhances training stability, sample efficiency, and final performance on mathematical reasoning tasks. More importantly, TOP-D introduces zero additional computational overhead, positioning itself as a promising alternative to the well-established OPD paradigm.

Introduction. On-Policy Distillation (OPD) [10] has rapidly emerged as the prevailing paradigm for post-training Large Language Models (LLMs) [18, 16, 21, 19, 1], in which the student learns from token-level signals provided by an external teacher. Mathematically, this is equivalent to reinforcement learning where the immediate reward is defined by the logarithmic probability ratio between the teacher and student policies. Consequently, the on-policy nature of OPD prevents the catastrophic forgetting of Supervised Fine-Tuning (SFT) [14], while its dense rewards overcome the sample inefficiency and instability of Reinforcement Learning with Verifiable Rewards (RLVR) (as summarized in Table 1). Despite its theoretical appeal, standard OPD suffers from severe optimization fragility in practice. The fundamental bottleneck of OPD lies in the inherent capacity gap between the target teacher and the student [5].

Discussion / Conclusion. In this paper, we introduce Trust Region Policy Distillation (TOP-D), a principled post-training paradigm resolving the optimization instability and sample inefficiency of standard On-Policy Distillation (OPD). Theoretically, TOP-D bounds gradient variance via an external proximal teacher and guarantees monotonic improvement with safe off-policy data reuse through internal trust region iterations. Empirically, it achieves massive performance gains on mathematical reasoning benchmarks, decisively outperforming standard OPD and competitive RLVR baselines across multiple model scales. By achieving these advancements without any additional computational overhead, TOP-D establishes a highly reliable, sample-efficient, and robust paradigm for aligning foundation models.