When an AI learns from a teacher model, is the teacher's token-by-token coaching essential, or does a flat penalty work as well?
Can constant penalties replace teacher-provided advantages in token supervision?
This explores whether the per-token 'how much better was this token than expected' signal that a teacher supplies during distillation can be swapped for a fixed penalty that pushes down tokens the student finds unlikely, and the corpus suggests that for on-policy distillation it largely can.
This explores whether the per-token signal a teacher supplies during distillation can be swapped for a fixed penalty on tokens the student finds unlikely. For on-policy distillation, the corpus suggests it largely can. One finding shows that Does on-policy distillation need a teacher at all? gets much of its gain from pushing probability away from low-likelihood tail tokens, and that a simple fixed penalty reproduces this benefit with no teacher at all. The same work reports that the teacher's fine-grained supervision adds substantial noise that students effectively ignore. The teacher's opinion of each token may be doing far less than assumed. Much of the benefit comes from the student discarding its own unlikely choices.
This fits a wider pattern: token-level learning signal is concentrated and redundant. Do high-entropy tokens drive reasoning model improvements? finds that only about 20% of tokens are high-entropy decision points, and training on just those matches or beats full updates. That is a different target (forking tokens rather than the low-probability tail), so the two findings shouldn't be merged. They do share a lesson, though. A blunt signal aimed at the right tokens can stand in for a detailed one applied everywhere.
The teacher's signal also has costs that a constant penalty avoids. Does richer teacher context hurt student generalization? shows that teachers conditioned on correct answers pass on confident, concise traces, which helps in-domain but hurts out-of-distribution problems that call for caution. Does teacher-refined data always improve student model performance? shows that teacher improvements beyond the student's learning frontier can make things worse, even when the data is objectively better. Sitting slightly in tension with the 'students ignore the teacher' result, these suggest students do inherit teacher style, sometimes to their detriment. A fixed penalty can't inject overconfidence. It also can't inject new knowledge, and the corpus never tests a case where the teacher knows something the student can't reach alone. So the result is about where distillation's gains come from, not proof that teachers are useless.
The corpus also offers a middle path: replace the external teacher with the student itself. Can environment feedback replace scalar rewards in policy learning? has the policy, when shown feedback about its own mistakes, act as its own process reward model, which yields dense token-level credit without an outside teacher. That is richer than a constant, but it points the same way, since the supervision can come from inside the student. Related work gets step-level granularity from only final-answer feedback, as in Can curriculum learning approximate expensive process supervision?. Another split of roles, Can rubrics and dense rewards work together without hacking?, uses dense token rewards to optimize while categorical rubrics gate what counts as valid. A fixed penalty would plausibly fit the shaping role there, but not the checking role.
Sources 7 notes
Research shows that on-policy distillation's gains come largely from pushing probability away from low-likelihood tokens, a benefit that a simple fixed penalty reproduces without any teacher. Teacher supervision contributes substantial noise that students ignore.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
Show all 7 sources
R3 progressively slides the reasoning start state backward from near-completion, creating a curriculum that reveals step-level failure modes using only outcome feedback. This achieves process supervision granularity without expensive human step annotations.
DRO shows that using rubrics to accept or reject rollout groups—rather than converting rubric scores into dense rewards—prevents reward hacking. This separation preserves the categorical strength of rubrics while letting token-level rewards optimize within valid answers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- On-Policy Self-Distillation without Any Supervision
- Self-distillation Enables Continual Learning
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning