Does on-policy distillation need a teacher at all?

The paper asks whether on-policy distillation (OPD) distills at all, and its answer is that the teacher may contribute less than the name implies. The authors report "substantial noise" in the teacher's supervision of student-generated trajectories, with prevalence that "increases with teacher scale." Yet the student is "insensitive to such noise," converging to comparable performance whether noisy supervision is kept or removed. Looking at what drives the gains, they find that "learning concentrates on low log-probability tokens" and that "using a single fixed negative advantage matches the performance of teacher-provided ones." The abstract's conclusion is that OPD "works largely by suppressing low log-probability tokens, which requires no teacher." The discussion states this more carefully: "much of OPD's improvement" can be reproduced without teacher supervision.

The mechanism the paper offers is a change of what the dense signal is doing. The teacher scores trajectories that are "inherently off-policy for it," so its token-level advantages are unreliable. The gain appears to come from pushing probability away from tail tokens, and a constant penalty on low-probability tokens does that as well as a teacher does. The proposed method, On-Policy Self-Adaptation (OPSA), scales the negative advantage by token-level entropy. It gives stronger signals at high-entropy positions, suppresses tail tokens, and "evenly" redistributes the freed mass among head tokens. The discussion reports that this sharpens the distribution where the model is already confident while preserving exploration "at high-entropy reasoning forks," and yields longer, more reflective trajectories.

This sits awkwardly against two neighbors. Does on-policy distillation actually expand student capability? also says teacher scale is not what matters, but it locates success in the fidelity of the guiding signal. This paper goes one step further: if a constant negative advantage does as well, the teacher's per-token fidelity is not what separates the runs, at least in the setting studied. The two agree that OPD reshapes sampling rather than adding capability. Can proximity between teacher and student fix distillation instability? repairs the teacher-student gap by constructing a better teacher. The excerpt here suggests a teacher may be dispensable for a large share of the benefit, though the two papers are not compared directly. The entropy-adaptive design also bears on Does policy entropy collapse limit reasoning performance in RL?, because OPSA sharpens low-entropy positions while claiming to keep high-entropy forks open. And Can environment feedback replace scalar rewards in policy learning? is a different route to teacher-free dense signal, one that still relies on environment feedback.

The excerpt does not say which models, teachers, benchmarks or training budgets were used. It gives no effect sizes and does not define how "noise" was measured or removed, and it does not say what OPSA is compared against when it reports "substantial improvements" in Avg@32 and Pass@32. The abstract's "matches" and the discussion's "much of" differ in strength, so the safe reading is that a teacher-free negative advantage recovers most, not necessarily all, of OPD's gain. What follows is a caution about attribution. Where OPD improves a student, part of the credit may belong to a generic tail-suppression effect rather than to knowledge transferred from the teacher, so a claimed teacher contribution needs a fixed-negative-advantage control before it counts.