A stronger AI can polish training data, but does that polish help every student model — or backfire on some?
Does teacher-style refinement of training data transfer equally to all student model distributions?
This explores whether using a stronger teacher model to rewrite or polish training data helps every student model the same way, or whether the benefit depends on how the refined data relates to what each student can already do.
No, the transfer is not equal. Teacher refinement helps only when it lands within what the student can absorb. Does teacher-refined data always improve student model performance? finds that teacher-refined data can hurt a student when it exceeds the student's learning frontier, even if the refined data is objectively higher quality. The proposed fix is for the student to filter refinements against its own statistical profile and keep only the compatible ones. "Better data" depends on who is reading it.
The transfer also carries the teacher's habits, not only its content. Does richer teacher context hurt student generalization? shows that teachers who see the correct answer and verifier output write confident, concise reasoning traces. Students inherit that style, including the loss of uncertainty language. That helps in-domain performance but hurts on out-of-distribution problems, where a model needs to sound cautious. So a student can absorb the refinement and still pick up a trait that doesn't travel. RL shows a parallel in a different setting: Does binary reward training hurt model calibration? shows that rewarding only correct answers pushes models toward confident guessing.
Other notes describe the same shape outside distillation, so these are analogies rather than direct tests. Do overly hard RLVR samples actually harm model capabilities? finds that training signal beyond what a model can reach gets learned as shortcuts, and those shortcuts contaminate abilities the model already had. Does staying close to the base model preserve learning ability? finds that staying close to the base distribution preserves the ability to keep learning, which fits the idea of filtering refinements by compatibility. Does RL training collapse format diversity in pretrained models? adds that which pretraining format wins out depends on model scale. That hints that students of different sizes may be primed to absorb different things, though the note doesn't test this for teacher data.
Two notes show ways around the mismatch. Can smaller models outperform their LLM teachers with enough data? found that small BERT students beat their LLM teachers once trained on enough teacher-labeled data. The student saw a broader range of inputs, and the teacher's predictions smoothed them, so the teacher acted as a source of signal rather than a ceiling. Can a stronger model lift a weaker one at test time without retraining? skips training entirely: a stronger model built inference-time harnesses that nearly doubled a weaker model's Theory-of-Mind scores. It did this mostly by moving unstable reasoning into deterministic code and task-specific routing.
The corpus has one direct note on teacher-refined data and student compatibility, and the rest is supporting evidence from neighboring problems. It has no head-to-head comparison of refinement across many student sizes or families. What it does suggest is that refined data should be matched to the student, and that scaffolding around the student is another way to pass on a teacher's strengths.
Sources 8 notes
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
Show all 8 sources
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
A stronger model built inference-time harnesses that nearly doubled weaker model performance on Theory-of-Mind benchmarks without retraining, primarily by moving unstable reasoning into deterministic code and task-specific routing rather than encouraging extended reasoning.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- AI4AI at Test-Time: Strong-to-Weak Capability Transfer via Harnesses
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training