No matter how good the teacher, an AI can only learn what it was already close to knowing.
How does student capacity limit what it can learn from teachers?
This explores how a student model's own capacity — its size, its existing skill envelope, its statistical profile — sets a hard ceiling on what teacher supervision can actually transfer, no matter how good the teacher is.
This explores how a student model's own capacity — not the teacher's quality — governs what supervision can transfer. The corpus keeps landing on the same surprising point: teaching mostly reorganizes what a student could already do, rather than adding genuinely new ability. On-policy distillation, for instance, turns out to be an exploration catalyst, not a capability expander — it steers a student toward correct reasoning paths inside its existing envelope, and a small high-fidelity teacher beats a giant vague one Does on-policy distillation actually expand student capability?. The corollary is blunt: objectively better teacher data can *hurt* if it exceeds the student's learning frontier, so students should filter refinements against their own statistical profile and keep only what's compatible Does teacher-refined data always improve student model performance?.
Why the ceiling exists gets concrete when you look at capacity as a measurable quantity. Models memorize up to roughly 3.6 bits per parameter, and only when that store fills does a phase transition into real generalization begin — capacity is a property of the individual model, not the training recipe When do language models stop memorizing and start generalizing?. In-weight factual recall is likewise formally bounded by parameter count Can models store unlimited facts without growing larger?. So there's a literal budget: a student can only absorb so much before new material overwrites old, which is exactly why teacher refinements past the frontier degrade rather than improve.
But capacity isn't only about raw size, and this is where the corpus gets interesting. Larger models learn rare things better not because they can *represent* what smaller ones can't, but because spare capacity weakens the gradients from common tasks, so rare features stop getting overwritten — the limit is interference, not expressivity Why do larger models learn rare tasks better?. That reframes 'small student can't learn X' as often really 'small student's gradients collide.' And several notes show the ceiling is lower than people assume: sub-2B models learn faithful, grounded, citation-backed reasoning from curriculum design alone Can small models learn to ground answers in context?, and models trained on child-scale data approach human grammatical competence Can language models learn grammar from child-scale data?. Capacity gates *what kind* of learning, and the right supervision format can unlock more of a small student than scale would suggest.
The teacher side has its own capacity logic. Learning requires information asymmetry — the teacher must know something (the answer, the verifier output) the student doesn't, or there's no corrective gradient at all Why does teacher-student information asymmetry enable learning signals?. But asymmetry cuts both ways: teachers conditioned on the correct answer produce confident, concise traces that a student inherits wholesale, trading away the uncertainty-expression it would need for out-of-distribution problems Does richer teacher context hurt student generalization?. And distance itself is a failure mode — a distant teacher blows up gradient variance, which is why building a *proximal* teacher inside a trust region turns unstable distillation into monotonic improvement Can proximity between teacher and student fix distillation instability?.
Here's the twist worth leaving with: capacity limits what a student can *learn*, but not what it can ultimately *outperform*. Walmart's small BERT cross-encoders beat their own LLM teachers once trained on enough teacher-labeled data — the student saw a broader input distribution, smoothed by the teacher's predictions, and generalized past what the teacher itself achieved Can smaller models outperform their LLM teachers with enough data?. So the honest frame isn't 'a small student is capped by the teacher.' It's that a student's capacity determines *how* it must be taught — proximal signal, compatible data, the right curriculum — and get that right and a modest student can exceed the teacher that trained it.
Sources 11 notes
On-policy distillation steers students toward correct reasoning paths within their existing capability envelope rather than raising the ceiling. Signal quality and diversity matter far more than teacher scale; a smaller teacher with high-fidelity guidance outperforms larger teachers without it.
Teacher-refined data degrades performance when it exceeds the student's learning frontier, even if objectively higher quality. Students should filter refinements using their own statistical profile to retain only compatible improvements.
GPT-family models have a measurable memorization capacity of approximately 3.6 bits-per-parameter. When this capacity fills, a phase transition triggers grokking—the shift from memorization to genuine generalization. This capacity is a property of individual models, not training algorithms.
A formal proof and experiments show in-weight memorization is bounded by model size, while tool-use enables unbounded factual recall through a simple circuit. In-weight finetuning also degrades general capability by overwriting prior knowledge.
Larger models succeed at rare tasks not because they can represent solutions smaller models cannot, but because abundant capacity weakens gradients on common tasks, preventing them from overwriting slowly-accumulating rare-task features. Data-mixture design may be cheaper than scaling.
Show all 11 sources
Sub-2B models trained on synthetic multi-hop QA can ground answers in passages, cite literal quotes, and abstain from confabulation. The OCC-RAG work shows faithfulness emerges from training curriculum design, not parameter count.
Models trained on ≤100 million words performed within a few percentage points of humans on grammatical acceptability tasks, suggesting syntactic competence doesn't require massive datasets. Data composition and curation mattered more than raw volume.
Social meta-learning requires information asymmetry—the teacher's access to correct answers or verifier output—to generate meaningful corrective signals. Without this asymmetry, teacher and student share identical uncertainty, making pedagogical correction impossible.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
Walmart's student cross-encoders outperformed their LLM teachers when trained on sufficiently large augmented datasets of teacher-labeled queries. The student's broader input distribution exposure, smoothed by teacher predictions, enabled better generalization than the teacher achieved.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Provable Benefits of In-Tool Learning for Large Language Models
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- How much do language models memorize?
- Trust Region Policy Distillation
- Bigger is not always better: The importance of human-scale language modeling for psycholinguistics
- Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey