When teaching an AI, does showing it thousands of different problems beat drilling deep into just a few?
How does prompt diversity compare to per-problem sampling depth in distillation?
This explores a distillation trade-off: whether it's better to spread teacher-generated training data across many different prompts (breadth) or to sample many solution traces from a smaller set of problems (depth per problem) — and what the corpus says about which buys you more.
This question pits two knobs against each other in distillation: how many *different* problems you show the student (prompt diversity) versus how *deeply* you sample the teacher on each one (traces per problem). The corpus doesn't have a head-to-head benchmark, but read laterally it suggests the two aren't symmetric — and that the most surprising result sits at the extreme of depth-over-breadth.
The strongest signal is that per-problem depth can be shockingly cheap and still work. Critique Fine-Tuning activates reasoning comparable to full RLVR using *a single problem* — the student learns not from breadth of prompts but from exposure to many contrasting solutions (correct vs. incorrect) on that one problem, critiqued by the teacher Can a single problem unlock reasoning through solution critique?. That reframes the question: if one problem sampled deeply and contrastively can unlock the capability, prompt diversity may matter less for *activation* than for *coverage* of the skill you're trying to transfer. Depth here isn't just repetition — it's variety of solutions within a problem, which is a different thing than variety of problems.
But depth only pays off when you can tell good traces from bad. Two notes converge here: step-level confidence filtering shows that a few high-quality traces beat a flood of them — trace quality matters more than trace quantity, and you can stop generating early Does step-level confidence outperform global averaging for trace filtering?. And weak-model committees show that sampling deeply amplifies coverage of correct answers but *cannot by itself select them* — you need an external soundness signal (tests, proofs, verifier output) to convert deep sampling into usable distillation data When can weak models match strong model performance?. So per-problem depth is only as good as your filter; unfiltered, it just piles up noise.
There's also a hidden cost to how you generate those deep samples. Teachers conditioned on the answer and verifier output produce confident, concise traces — and students inherit that confidence, gaining in-domain sharpness but *losing* out-of-distribution robustness because the uncertainty got trained out Does richer teacher context hurt student generalization?. This is where prompt diversity earns its keep: breadth across problem types is a hedge against a student that's brittle outside the narrow region it was sampled deeply on. Depth sharpens; breadth generalizes.
The deepest cross-cutting lesson comes from the diversity-and-search literature. When students later feed into search or repeated sampling at inference, training should *maximize diversity of competent solutions* rather than collapse to one mode — entropy-collapsed policies literally cannot reach problems that mode-diverse ones can Should training maximize diversity when models feed into search?. And outcome-based sharpening leaks: reducing diversity on solved problems quietly reduces it on unsolved ones too Does outcome-based RL diversity loss spread across unsolved problems?. The takeaway isn't 'breadth vs. depth' as a fixed ratio — it's that depth without a selection signal and without preserved solution diversity buys you a confident, narrow student, while breadth is what keeps that student from falling off a cliff the moment the problem looks unfamiliar.
Sources 6 notes
Critique Fine-Tuning achieves reasoning activation comparable to RLVR using only one problem and teacher-generated critiques of varied solutions, with no reinforcement learning. This demonstrates that exposure to correct versus incorrect reasoning on a specific problem is the sufficient activation signal.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Sampling alone amplifies coverage but cannot select correct solutions. Reliable performance matching requires external soundness signals—tests, proofs, or type checks—that convert latent correct proposals into actual selections.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Vector Policy Optimization trains models to emit varied competent solutions rather than converging to one answer. This unlocks search procedures like evolutionary algorithms to explore and combine modes, solving problems that entropy-collapsed policies cannot reach at all.
Show all 6 sources
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Outcome-based Exploration for LLM Reasoning
- Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Rethinking Thinking Tokens: LLMs as Improvement Operators
- Deep Think with Confidence
- Agentic Systems as Boosting Weak Reasoning Models
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?