How can distillation preserve uncertainty expression instead of optimizing it away?
This explores whether distillation — training a smaller or cleaner 'student' model on a 'teacher' model's outputs — has to flatten away the hedges, second-guessing, and signals of doubt that the teacher expressed, or whether those signals can be deliberately protected.
This explores whether distillation has to optimize away a model's expressions of uncertainty — the hedges and second-guessing — or whether those signals can be preserved on purpose. The corpus is surprisingly pointed on this: the loss isn't a side effect, it's what the usual objective rewards. Teachers conditioned on the correct answer and verifier output produce confident, concise traces, and students faithfully inherit that confident style — trading out-of-distribution robustness for polished in-domain performance Does richer teacher context hurt student generalization?. Self-distillation shows the same pattern from the inside: it quietly strips the 'Wait' and 'Hmm' tokens that mark a model pausing over a shaky reasoning step, and removing those markers removes the model's ability to catch and correct itself on unfamiliar problems Does self-distillation harm mathematical reasoning performance?.
The deeper diagnosis is that this is a measurement blind spot. Post-training objectives are good at one thing — steering toward correct answers — and anything they don't explicitly measure, like uncertainty-aware reasoning style, is left unprotected and gets eroded as a free variable Can post-training objectives preserve reasoning style alongside correctness?. So the first answer to 'how can distillation preserve uncertainty' is: make uncertainty something the objective actually scores, rather than collateral. Notice too that the confidence which gets baked in isn't always earned — post-trained models run 3-4x lower output entropy on their own generated text, a self-recognition reflex that lowers expressed doubt without any real change in what the model knows Why do models produce less uncertain outputs on their own text?.
A second route sidesteps the weights entirely. Proxy-tuning applies the alignment shift at decoding time and leaves the base model's parameters untouched, closing most of the alignment gap while *outperforming* direct fine-tuning on knowledge tasks — because direct fine-tuning corrupts knowledge in the lower layers, whereas a decoding-time nudge mostly touches style and reasoning Can decoding-time tuning preserve knowledge better than weight fine-tuning?. If you never overwrite the weights that hold the model's calibrated sense of doubt, you can't optimize that doubt away. Relatedly, you can build the capacity to hold uncertainty into the architecture: making latent reasoning stochastic lets a model represent a distribution over solutions instead of collapsing to one confident guess, so ambiguity survives as a first-class state rather than getting rounded off Can stochastic latent reasoning help models explore multiple solutions?.
What makes preservation worth the trouble is that the uncertainty signal is genuinely useful downstream, not just decorative. Calibrated token-probability uncertainty beats elaborate adaptive-retrieval heuristics at deciding when a model should go look something up — the model's own self-knowledge is more reliable than external machinery, and cheaper Can simple uncertainty estimates beat complex adaptive retrieval?. Confidence also predicts robustness: highly confident models resist prompt rephrasing while low-confidence ones swing wildly, so confidence carries real information about when to trust an output Does model confidence predict robustness to prompt changes?. And step-level confidence catches reasoning breakdowns mid-trace that a single global confidence score papers over Does step-level confidence outperform global averaging for trace filtering?. Distill the global confidence and you lose exactly the local signal that flags where the reasoning went wrong.
The thread tying these together: a confident-looking model isn't a calibrated one. Deterministic settings produce the same answer every time without making that answer reliable — it's still one draw from a distribution Does setting temperature to zero actually make LLM outputs reliable?. Distillation that optimizes for crisp, confident traces is manufacturing that false steadiness at scale. Preserving uncertainty, then, means three concrete moves the corpus points to: score uncertainty-style in the training objective so it isn't silently dropped, shift behavior at decoding time instead of overwriting calibrated weights, and keep uncertainty representable in the model's reasoning rather than collapsing it to a point estimate. The thing you didn't know you wanted to know: the hedges and 'Wait' tokens aren't verbal tics — they're the load-bearing scaffolding for self-correction, and a smoother, more confident student is often a more brittle one.
Sources 10 notes
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.
Self-distillation reduces performance in mathematical reasoning by eliminating epistemic markers like "Wait" and "Hmm" tokens that flag flawed reasoning paths. These tokens enable self-correction on out-of-distribution problems, so removing them sacrifices robustness for confident brevity.
Research shows that post-training objectives faithfully guide models toward correct answers yet simultaneously suppress unmeasured behaviors like epistemic verbalization. Single-objective optimization creates blind spots where stylistic features critical to generalization are unprotected.
Post-trained models produce 3-4x lower output entropy on their own generations, driven by an internal representation of input surprise that causally modulates confidence. This implicit self-recognition signal appears without being verbalized, encoded directly in the output distribution.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
GRAM replaces deterministic latent updates with stochastic sampling, enabling models to represent distributions over solutions rather than single predictions. This allows handling of ambiguous problems and multiple valid strategies that deterministic designs cannot represent.
Calibrated token-probability uncertainty consistently beats multi-call adaptive retrieval on single-hop tasks and matches performance on multi-hop, using a fraction of the LM and retriever calls. The model's self-knowledge proves more reliable than external heuristics for deciding when to retrieve.
ProSA found that when models are highly confident, they resist prompt rephrasing; low confidence causes major output swings. Larger models, few-shot examples, and objective tasks all correlate with higher confidence and greater robustness.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Fixed seeds and zero temperature replicate the same output repeatedly, but that output remains one draw from the model's probability distribution. McDonald's omega testing across 100 repetitions reveals that consistency does not equal reliability.