When training a smaller AI, a slightly better teacher often beats a much smarter one — proximity matters more than power.
Why does teacher-student proximity matter more than absolute teacher strength?
This explores why the *gap* between a teacher model and the student it trains matters more than how strong the teacher is in absolute terms — and why a close, compatible teacher often beats a bigger, better one.
This explores why the *gap* between teacher and student matters more than raw teacher strength in knowledge distillation. The corpus keeps landing on the same counterintuitive point: what a student can actually absorb is bounded by its own current position, so a teacher that's too far ahead pours signal into a space the student can't reach. Teacher-refined data that's objectively higher quality still *degrades* the student when it exceeds the student's learning frontier — the fix is for the student to filter refinements against its own statistical profile and keep only what's compatible Does teacher-refined data always improve student model performance?. Distance, not quality, is the failure axis.
The mechanism behind this shows up cleanly in on-policy distillation. When you build a *proximal* teacher bounded by a trust region instead of distilling toward a distant target, you control gradient variance and get monotonic improvement — with zero extra compute Can proximity between teacher and student fix distillation instability?. A distant teacher creates high-variance gradients that make training unstable; proximity is what tames them. This rhymes with a broader finding about staying close to your own base: low KL drift from the starting distribution preserves plasticity, so the model keeps learning downstream, while aggressive parameter-only updates stall when the domain shifts Does staying close to the base model preserve learning ability?. In both cases, staying near a reachable reference point is what keeps learning healthy.
The most direct answer to 'why not just use a bigger teacher?' is that scale often isn't doing the work you think it is. On-policy distillation turns out to be an *exploration catalyst, not a capability expander* — it steers the student toward correct reasoning paths inside its existing envelope rather than raising the ceiling, and a smaller teacher with high-fidelity, diverse guidance beats a larger teacher without it Does on-policy distillation actually expand student capability?. Signal quality and match beat scale. You can even see the inversion: Walmart's small BERT cross-encoders *outperformed* their LLM teachers once trained on enough teacher-labeled data, because the student's broader input exposure generalized better than the teacher ever did Can smaller models outperform their LLM teachers with enough data?.
But proximity has a floor as well as a ceiling — the teacher can't be *identical* to the student either. Learning requires an information asymmetry: the teacher's privileged access to correct answers or verifier output is what generates the corrective gradient in the first place. Collapse that gap entirely and teacher and student share the same uncertainty, making pedagogical correction impossible Why does teacher-student information asymmetry enable learning signals?. So the real target is a *sweet spot*: close enough to be reachable, asymmetric enough to have something to teach. And that asymmetry can be double-edged — a teacher conditioned on answers produces confident, concise traces the student inherits, which sharpens in-domain accuracy but suppresses the epistemic caution needed for out-of-distribution problems Does richer teacher context hurt student generalization?.
Step back and this is really a special case of a deeper pattern in self-improvement: reliable gains come from an external anchor that is *close but informative* — a past model version, a judge, a verifier, a user correction — rather than from a distant oracle or from pure self-reference, which collapses into circularity Can models reliably improve themselves without external feedback?. The surprising takeaway: the best teacher isn't the smartest one available — it's the one positioned just ahead of where the student already stands.
Sources 8 notes
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Show all 8 sources
Social meta-learning requires information asymmetry—the teacher's access to correct answers or verifier output—to generate meaningful corrective signals. Without this asymmetry, teacher and student share identical uncertainty, making pedagogical correction impossible.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Pure self-improvement stalls due to the generation-verification gap, diversity collapse, and reward hacking. Reliable improvement methods succeed by smuggling in external anchors: past model versions, third-party judges, user corrections, or tool feedback.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Self-distillation Enables Continual Learning
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Trust Region Policy Distillation
- Demystifying On-Policy Distillation: Roles, Pathologies, and Regulations
- Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
- SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training