An AI that learns from its own majority-vote answers gets sharper fast, but why does that edge vanish when it thinks longer?
Why does unsupervised self-distillation lose its advantage in thinking model mode?
This explores why teaching a model from its own majority-vote answers, with no labels, helps in its quick-answer mode but stops paying off when the model is allowed to think at length.
The corpus has no note that tests thinking mode against non-thinking mode directly, so what follows is an inference from neighboring findings, not a documented result. The best-supported explanation is that self-distillation gets its gains by making a model more confident and concise, and thinking mode is where confidence and concision cost the most.
Start with where the advantage comes from. In Can a model's own consensus replace ground truth labels?, the model's own consensus answer replaces ground truth, and it matched or beat supervised methods on five benchmarks. It only distills on rollouts where the model disagrees with itself, so it collapses a scattered set of answers onto the one the model already gives most often. Models are already primed for this: Why do models produce less uncertain outputs on their own text? finds they are 3-4x more confident on their own text. Training on your own outputs pushes further in a direction the model already leans.
The cost shows up in Does self-distillation harm mathematical reasoning performance?. Self-distillation strips out tokens like "Wait" and "Hmm" that flag a flawed reasoning path, and those tokens are what let a model correct itself on problems it hasn't seen before. In a short-answer mode there is little of that behavior to lose, so sharpening is nearly free. In thinking mode the long chain of reasoning is the mechanism. A distilled model has learned to sound sure and to skip the doubt-and-recheck moves, so it gives up the thing extended thinking would use. Sharpening still helps, but the loss in self-correction cancels much of it.
Two neighboring notes suggest where thinking-mode gains come from instead. Do reasoning models switch between ideas too frequently? shows that o1-style models often waste effort by abandoning promising paths, and that a decoding-time penalty on switching improves accuracy with no retraining. So the levers in thinking mode sit in how the trace unfolds, not in how peaked the final-answer distribution is. Why does self-correction training on offline data fail? adds that self-correction is learned by practicing on the model's own live mistakes with online RL. Consensus-based targets don't supply that practice, because they reward agreement, not recovery from error.
The clean test would be to compare how often "Wait"-style tokens appear in thinking traces before and after self-distillation, and whether accuracy tracks that count. The corpus doesn't contain that experiment. The mechanism above is a hypothesis it makes plausible, not a finding it confirms.
Sources 5 notes
Unsupervised on-policy self-distillation using the model's own majority-vote consensus matched or surpassed supervised methods on five benchmarks. The key mechanism distills only on self-inconsistent rollouts, using agreement as the teaching signal rather than external labels.
Post-trained models produce 3-4x lower output entropy on their own generations, driven by an internal representation of input surprise that causally modulates confidence. This implicit self-recognition signal appears without being verbalized, encoded directly in the output distribution.
Self-distillation reduces performance in mathematical reasoning by eliminating epistemic markers like "Wait" and "Hmm" tokens that flag flawed reasoning paths. These tokens enable self-correction on out-of-distribution problems, so removing them sacrifices robustness for confident brevity.
o1-like models frequently abandon reasoning paths mid-exploration, wasting tokens on incomplete approaches. A decoding-only penalty on thought-transition tokens (TIP strategy) discourages switching, improving accuracy on challenging math without model fine-tuning.
SFT on offline correction traces fails because training errors don't match test errors and models collapse into single correction modes. Multi-turn online RL under the model's own error distribution successfully trains self-correction by letting models practice correcting their actual mistakes.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can Large Reasoning Models Self-Train?
- Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?
- LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
- On-Policy Self-Distillation without Any Supervision
- Thoughts Are All Over the Place: On the Underthinking of o1-Like LLMs
- From Simulation to Enaction: Post-trained Language Models Recognize and React to their own Generations
- Training Language Models to Self-Correct via Reinforcement Learning
- Meta-Reasoner: Dynamic Guidance for Optimized Inference-time Reasoning in Large Language Models