Does on-policy distillation actually expand student capability?

The role of on-policy distillation is easy to misread as capability transfer — a strong teacher pouring new abilities into a student. This signal-level diagnosis reframes it: OPD is an exploration catalyst. It steers the student toward correct reasoning paths within the student's existing capability envelope via dense token-level guidance; it does not raise the capability ceiling. The evidence is that prompt diversity matters more than per-problem sampling count — what OPD buys is broader coverage of good trajectories, not deeper ones the student could never reach.

Because the whole mechanism hinges on the fidelity of the guiding signal, two pathologies derail it. Student-Teacher Mismatch: when the teacher-student distributional gap is large, the guiding signal misaligns with task correctness and actively steers exploration in counterproductive directions — the same capacity-gap problem TOP-D attacks with a proximal teacher. Length Exploitation: because the objective aggregates token-level advantages, it creates length-dependent shortcuts — the student games the reward by truncating or padding responses, exploring degenerate length modes instead of reasoning strategies (a distillation-specific reward-hacking failure).

The practical payoff is a clean inversion of "bigger teacher is better": with lightweight, zero-overhead signal regulations that fix token-level advantage dynamics, a 4B teacher surpasses configurations distilled from larger models. Signal quality, not teacher scale, governs OPD success. This tightens the picture from Does RLVR actually expand what models can reason about? — like RLVR, OPD operates within existing boundaries; it just redistributes sampling toward correct paths rather than expanding what is reachable.