How does inference variance differ from training entropy collapse?
This explores why reasoning models break in two different ways — losing diversity during training (entropy collapse) versus producing wildly inconsistent answers at deployment (inference variance) — and why those are not the same problem.
This explores why reasoning models break in two different ways — too little variety during training, too much during use — and why fixing one doesn't fix the other. The cleanest framing in the corpus calls these *dual* failures of the same underlying thing: the balance between exploring new options and exploiting known-good ones. They just show up at opposite ends of the pipeline and on different clocks Why do reasoning models fail differently at training versus inference?.
Entropy collapse is a *training* pathology. As a model is optimized with reinforcement learning, its output distribution narrows — it grows confident and stops trying alternatives. There's even a predictable ceiling to it: performance saturates as policy entropy heads toward zero, and once the model stops exploring, it simply can't get better no matter how much more you train Does policy entropy collapse limit reasoning performance in RL?. This collapse isn't random; it concentrates on the small minority of high-entropy "forking" tokens where reasoning decisions actually get made — squeeze those and you've squeezed the learning signal Do high-entropy tokens drive reasoning model improvements?. It also shows up as format collapse, where RL quietly funnels the model onto a single dominant answer style and suppresses the others it knew from pretraining Does RL training collapse format diversity in pretrained models?.
Inference variance is the opposite complaint, and it happens after training is done. Here the model produces *too much* spread — ask the same question a few times and get answers of wildly different quality. The important insight is that a training-time fix (entropy bonuses, keeping diverse critiques alive) does nothing for this, and an inference-time fix does nothing for collapse. They're separate control loops that each have to be managed on their own terms Why do reasoning models fail differently at training versus inference?. So "add more entropy" is not a universal cure — depending on which failure you have, more entropy is either the medicine or the disease.
What ties the two together is that both are really about a model's relationship to its own uncertainty. Post-trained models actually track how surprising their input is and lower their output entropy on text they generated themselves — an implicit, unverbalized confidence signal baked into the distribution Why do models produce less uncertain outputs on their own text?. That same machinery can be miscalibrated from either side: train on impossible problems and the model collapses onto degenerate shortcuts that masquerade as confidence Do overly hard RLVR samples actually harm model capabilities?, or distill from an over-informed teacher and the student inherits false certainty that looks fine in-domain but shatters into bad variance the moment it meets out-of-distribution problems Does richer teacher context hurt student generalization?.
The thing worth carrying away: confidence and diversity aren't virtues or vices on their own. Collapse is the failure of a model that became too sure during training; variance is the failure of a model that stays unsure at the wrong moments during use — and because they live at different points in the lifecycle, you have to diagnose which one you're actually looking at before reaching for a fix.
Sources 7 notes
Both failures stem from failed exploration-exploitation balance but occur at different timescales requiring structurally distinct interventions. Training-time fixes (entropy bonuses, critique diversity) cannot prevent inference-time variance inflation, and vice versa; both loops must be managed independently.
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
Only ~20% of tokens exhibit high entropy as pivotal reasoning decision points; RLVR primarily adjusts these forking tokens. Training exclusively on them matches or exceeds full-gradient performance, revealing that the minority carries the learning signal.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Post-trained models produce 3-4x lower output entropy on their own generations, driven by an internal representation of input surprise that causally modulates confidence. This implicit self-recognition signal appears without being verbalized, encoded directly in the output distribution.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Teachers conditioned on correct answers and verifier output produce confident, concise traces that students inherit. This style suppresses uncertainty expression, optimizing in-domain performance while degrading generalization to out-of-distribution problems that require epistemic caution.