What filtering criteria best identify student-compatible refinements from teacher models?
This explores how a smaller 'student' model should decide which improvements from a larger 'teacher' model to actually keep — the criteria that separate refinements a student can absorb from ones that look better but make it worse.
This explores how a smaller 'student' model should decide which improvements from a larger 'teacher' model to actually keep — and the surprising answer the corpus gives is that *objective* quality is the wrong filter. The cleanest result is that teacher-refined data can degrade a student even when it is genuinely higher quality, because it sits past the student's 'learning frontier.' The proposed criterion is the student's own statistical profile: keep refinements that fall inside what the student can already represent, discard the ones that don't, regardless of how good they look on paper Does teacher-refined data always improve student model performance?. So 'best' here means *compatible*, not *best in absolute terms*.
A second criterion is about the teacher's conditioning, not just the data points. When a teacher is fed the correct answer and verifier output up front, it produces confident, short traces — and the student inherits that confidence, including its suppression of uncertainty. That buys strong in-domain accuracy at the cost of out-of-distribution robustness Does richer teacher context hurt student generalization?. A good filter therefore screens not only for difficulty but for *style*: refinements that teach calibrated hedging may generalize better than refinements that teach slick certainty. This connects to a quieter danger — students can absorb the teacher's shortcuts. Several notes show models that look like they're reasoning but are really template-matching or defaulting conservatively Do fine-tuned language models actually learn optimization procedures? Are models actually reasoning about constraints or just defaulting conservatively? Do large language models actually perform iterative optimization?, so a refinement that improves the in-domain score might just be transmitting a memorized pattern the student can't extend.
The format of the signal turns out to be a filtering lever too. For small models, the most learnable refinements aren't just polished correct answers but *paired* correct-and-incorrect examples — DPO on a teacher's preference pairs beats plain supervised fine-tuning precisely because the negative examples target the rigid format failures students fall into Can small models match large models on function calling?. Relatedly, breaking quality into named attributes (clarity, relevance, specificity) and filtering refinements per-attribute outperforms filtering on a single global score Can models learn to ask genuinely useful clarifying questions?, and teaching with an explicit framework rather than raw labeled examples is what lets criteria transfer to new cases at all Can models learn argument quality from labeled examples alone?. The lesson: decompose what you're filtering for, and prefer refinements that carry the *reason*, not just the verdict.
There's a counterweight worth knowing. None of this means students are capped below teachers — with enough teacher-labeled data across a broad input distribution, student cross-encoders have *exceeded* their LLM teachers, because the student saw a wider slice of inputs smoothed by teacher predictions Can smaller models outperform their LLM teachers with enough data?. So coverage of the input space can be a filtering criterion in its own right: breadth of teacher-labeled examples beats depth of refinement on any single example.
The thing you might not have known you wanted to know: be careful about *who chooses* the refinements. If you lean on a model-as-judge to score teacher outputs, that judge is itself exploitable — authority cues and rich formatting fool LLM judges with zero-shot attacks Can LLM judges be fooled by fake credentials and formatting? — and the deeper bound is that no model can validate its own improvements without an external check, because of the gap between generating an answer and verifying it What stops large language models from improving themselves? Can model confidence work as a reward signal for reasoning?. The best filtering criterion is only as trustworthy as the verifier applying it.
Sources 12 notes
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Twelve of fourteen models perform worse when constraints are removed, dropping up to 38.5 percentage points. Models appear to reason correctly by defaulting to harder options, not by actually evaluating constraints.
Research shows LLMs cannot perform iterative procedures in latent space. They recognize optimization problems as template-similar and emit plausible-looking but incorrect values, a failure mode that persists across model scale and training approaches.
Small models fine-tuned via DPO on correct and incorrect function-calling examples from a large teacher model achieve high accuracy on logical and mathematical tasks. DPO's explicit negative examples directly target the rigid output format failures where SFT alone underperforms.
The ALFA framework breaks down question quality into theory-grounded attributes (clarity, relevance, specificity) and trains models on 80K attribute-specific preference pairs. Attribute-specific optimization outperforms single-score training, especially in clinical reasoning where asking the right clarifying question directly impacts decision quality.
Fine-tuning on labeled examples fails to transfer quality criteria to new argument types. Models learn surface patterns rather than principled criteria. Explicit instruction using frameworks like RATIO or QOAM significantly improves performance and generalization.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Research identified four evaluation biases in LLM judges, with authority and beauty biases being semantics-agnostic and trivially exploitable through fake references and formatting—zero-shot attacks requiring no model access or optimization.
Self-improvement in LLMs is formally bounded by the generation-verification gap, meaning every reliable fix requires something external to validate and enforce it. Models cannot escape this constraint through metacognition alone.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.