Does teacher-refined data always improve student model performance?
Explores whether higher-quality training data from teacher models uniformly benefits student models, or if compatibility with the student's current learning state matters for effective instruction.
Standard instruction tuning improvement pipelines assume: teacher refines training data → student trains on refined data → student improves. Selective Reflection-Tuning challenges this with a compatibility argument: data quality is relative to the student, not absolute. A response "improved" by a GPT-4 teacher may introduce knowledge complexity or reasoning patterns that conflict with the student's current knowledge state — producing degraded training signal despite being objectively higher quality.
The fix: after teacher refinement, have the student model evaluate each refined sample and decide whether to incorporate it. The student uses its own statistical profile as the selection criterion — what it finds tractable and useful given its current weights. Teacher-refined data the student can't process effectively is filtered out; compatible refinements are retained.
The underlying argument is metacognitive: the appropriate training signal for a model at capability level T is not the best possible response in absolute terms but the best response compatible with the model's current learning frontier. Overshoot in data quality creates a mismatch analogous to teaching advanced calculus before arithmetic is solid — the instruction is correct but the student can't absorb it.
This adds a dimension to the SFT quality literature. Correctness of training targets is necessary but not sufficient — compatibility with the specific student's current distribution is equally required. A data-quality pipeline that doesn't account for student compatibility will produce inconsistent results across different model sizes, initializations, and training stages.
Connects to Does supervised fine-tuning actually improve reasoning quality?: both identify SFT quality failures; this paper adds that even "better" data in absolute terms can degrade performance if the student-compatibility dimension is ignored.
Teacher benchmark scores don't predict teaching effectiveness (OpenThoughts): In SFT data curation for reasoning models, QwQ-32B outperforms DeepSeek-R1 as a teacher despite scoring lower on target reasoning benchmarks. This extends the student-compatibility argument: even the teacher dimension is not just about absolute quality. A weaker-performing model may produce responses whose reasoning patterns are more compatible with the student's learning frontier. Additional findings: quality source selection beats diversity (top 1-2 question sources > top 8-16), difficulty-based and response-length filtering outperform embedding-based or fastText filters, and sampling 16x answers per question is an effective scaling strategy — increasing dataset size 16x through multi-answer sampling drives significant gains.
Inquiring lines that read this note 66
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
How do self-generated feedback mechanisms enable effective model learning?- Does extended exoskeleton use eventually produce meaningful skill transfer?
- How does a challenger's escalating difficulty function as curriculum?
- Does curriculum-based training keep small models perpetually at their learning edge?
- How do self-generated preference pairs from a strong teacher compare to human feedback?
- Can curriculum degradation of document quality accelerate policy learning?
- Why do proprietary models improve with training while open-source models decline?
- How does distributional distance from pre-training relate to model difficulty?
- What makes utility-weighted training backfire in machine learning systems?
- Does pretraining data size matter less than base model scale for finetuning?
- Which finetuning method works best across different task and data regimes?
- How do finetuning and pretraining improvements differ in their effects on model capabilities?
- What trade-offs emerge between training objectives and model reliability?
- Can curated demonstrations compensate for smaller or simpler training environments?
- Why does curriculum order matter when information theory says data order is irrelevant?
- Why does self-generated training data outperform externally sourced data?
- Why does self-generated training data outperform externally curated domain examples?
- Can the serving loop itself become the primary training data source?
- How does training data distribution determine what models can learn?
- What training data contamination rates threaten model safety most practically?
- Does self-generated training data reduce a model's capability diversity?
- What makes policy self-distillation more effective than external teacher distillation?
- Why does the same training data produce different gains across models?
- Does selecting examples from multiple complexity levels outperform selecting only high-quality examples?
- Why do easy training examples contribute less to model generalization than hard ones?
- Can selecting the right data subset outperform training on everything?
- How much task-similar finetuning data does test-time training actually need?
- How do difficulty metrics relate to the true value of training examples?
- Why do adaptive curriculum schemes outperform static difficulty filters?
- How does difficulty-adaptive curriculum learning change which samples get selected during training?
- How does the optimal difficulty band shift as the model's capabilities improve during training?
- Why does training data format matter more than domain content?
- Does training data format matter more than who generates it?
- Can gradient-based influence scores beat difficulty metrics for identifying valuable training data?
- When does knowledge distillation produce student models superior to teachers?
- Why do weaker models generate better training data than stronger models?
- Why do weaker teacher models sometimes produce better training signals than stronger ones?
- What filtering criteria best identify student-compatible refinements from teacher models?
- Can self-training drift be prevented by applying student compatibility filtering?
- How does information asymmetry between teacher and student create the learning signal?
- Can teachers trained under uncertainty constraints distill better generalizing students?
- Why does information asymmetry between teacher and student enable effective feedback learning?
- Why does teacher-student proximity matter more than absolute teacher strength?
- What makes student-teacher distributional mismatch derail on-policy distillation?
- How does student capacity limit what it can learn from teachers?
- Can signal quality regulations help smaller teachers outperform larger ones?
- Why should we ignore bits where teacher and student already agree?
- Why does curriculum learning with tight budgets beat fixed-budget approaches?
- Do sample-level similarities between pretraining and downstream tasks explain the frequency effect?
- How does the Learning Law explain why all examples should contribute equally?
- What makes training data quality more important than quantity for reasoning?
- Why do students learn better from explanations than from solving problems from scratch?
- Does prompt performance vary by how well training data covers the domain?
- Do text-space skills transfer learning across different frontier models?
Related concepts in this collection 2
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does supervised fine-tuning actually improve reasoning quality?
While SFT boosts final-answer accuracy, does it degrade the quality and informativeness of the reasoning steps that justify those answers? This matters for high-stakes domains requiring auditable decision-making.
both identify SFT quality failures; student-compatibility is a missing dimension in the data-quality literature
-
How quickly do errors compound during model self-training?
When LLMs train on their own outputs without verification, do small mistakes amplify exponentially? This matters because it determines whether unsupervised self-improvement is even feasible.
teacher refinement without compatibility filtering is structurally vulnerable to the same drift as unchecked self-training
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- Subliminal Learning: Language models transmit behavioral traits via hidden signals in data
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
- AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Task Contamination: Language Models May Not Be Few-Shot Anymore
- AI Meets the Classroom: When Does ChatGPT Harm Learning?
Original note title
teacher-refined instruction data requires student-model selection because refinement compatibility depends on the student's current distribution