Does a bigger AI teacher train a student better, or does the quality of its guidance matter more?
Does teacher scale matter for on-policy distillation success?
This explores whether using a bigger, more capable teacher model makes on-policy distillation (where the student learns from feedback on its own generated outputs) work better, or whether something else about the teacher matters more.
This explores whether a bigger teacher makes on-policy distillation work better. The corpus says scale is not the main driver. One note finds that signal quality and diversity matter far more than teacher scale, and that a smaller teacher giving high-fidelity guidance can beat a larger teacher that doesn't Does on-policy distillation actually expand student capability?. The same note explains why: on-policy distillation mostly steers a student toward correct reasoning paths it could already reach, so it doesn't raise the ceiling. A bigger teacher has little extra to offer when the student's own ceiling is the limit.
A bigger teacher can also make things worse, because distance is a problem. If the teacher sits far from the student, the training signal gets noisy and unstable. The TOP-D method builds a deliberately close teacher inside a trust region, which the authors say guarantees steady improvement, and it beats standard distillation at no extra compute cost Can proximity between teacher and student fix distillation instability?. A related finding from off-policy data points the same way. Teacher-refined data hurts when it exceeds the student's learning frontier, even when it is objectively better, so students do best filtering for refinements that fit their own profile Does teacher-refined data always improve student model performance?. In both cases, a teacher that is well matched to the student beats one that is simply more capable.
Some findings ask whether the teacher does much at all. One analysis finds that much of the gain comes from pushing probability away from unlikely tokens, and a simple fixed penalty reproduces this without any teacher. It also finds that teacher supervision adds substantial noise the student largely ignores Does on-policy distillation need a teacher at all?. Another shows a model can distill from its own majority-vote consensus, with no larger model involved, and match or beat supervised methods on five benchmarks Can a model's own consensus replace ground truth labels?. If a model's own agreement can act as the teacher, scale is clearly not the essential ingredient.
What a teacher passes on matters more than its size. Teachers given the correct answer and verifier output write confident, concise traces, and students inherit that style. They score well in-domain but generalize worse on out-of-distribution problems that call for caution Does richer teacher context hurt student generalization?. The student can also end up ahead of the teacher. Walmart's small BERT cross-encoders outperformed their LLM teachers after training on enough teacher-labeled, augmented data Can smaller models outperform their LLM teachers with enough data?. That setup is off-policy, so it doesn't prove the same holds for on-policy distillation. The on-policy method itself (students generate sequences that the teacher scores, which removes the mismatch between training and inference) is described in Can distillation work on the student's own generated sequences?. The corpus doesn't directly test a scale sweep across teachers of different sizes with everything else held fixed, so its answer is that scale is secondary to closeness, signal quality and diversity.
Sources 8 notes
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Research shows that on-policy distillation's gains come largely from pushing probability away from low-likelihood tokens, a benefit that a simple fixed penalty reproduces without any teacher. Teacher supervision contributes substantial noise that students ignore.
Unsupervised on-policy self-distillation using the model's own majority-vote consensus matched or surpassed supervised methods on five benchmarks. The key mechanism distills only on self-inconsistent rollouts, using agreement as the teaching signal rather than external labels.
Show all 8 sources
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Generalized Knowledge Distillation (GKD) fixes the train-inference mismatch by having students generate sequences that the teacher then scores, aligning the training distribution with the student's actual inference behavior. This on-policy approach also composes cleanly with RL fine-tuning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- On-Policy Self-Distillation without Any Supervision
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Self-distillation Enables Continual Learning
- Trust Region Policy Distillation
- Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
- Reinforcement Learning via Self-Distillation