Does policy entropy collapse represent the main bottleneck in reasoning-focused RL scaling?
This explores whether the loss of exploratory diversity during RL training (policy entropy collapse) is *the* ceiling on reasoning RL — or just one bottleneck among several the corpus identifies.
This explores whether policy entropy collapse — the way RL training narrows a model's exploration until it keeps sampling the same few reward-maximizing strategies — is the dominant ceiling on reasoning RL, or one constraint competing with others. The corpus gives entropy collapse a strong, even quantified, case for being the bottleneck: there's an empirical law where reasoning performance saturates as policy entropy approaches zero, and interventions that deliberately preserve exploratory capacity (Clip-Cov, KL-Cov, GPPO) push the ceiling back up Does policy entropy collapse limit reasoning performance in RL?. What makes this more than a single-paper claim is that the same mechanism shows up in a different domain: RL training on search agents compresses behavioral diversity exactly the way it does in reasoning, and supervised fine-tuning on diverse demonstrations is what restores exploration breadth Does reinforcement learning squeeze exploration diversity in search agents?. So entropy collapse is at least a *general* failure mode, not a quirk of one benchmark.
But the corpus also suggests the bottleneck moves as training progresses, which complicates the 'main bottleneck' framing. RL training appears to run in two phases: first, execution correctness drives the gains; then strategic planning becomes the constraint, with planning-token entropy actually *rising* while execution entropy stabilizes Does RL training follow a predictable two-phase learning sequence?. That's a subtle but important wrinkle — late in training, you don't necessarily want entropy to keep falling everywhere; you want it concentrated in the right place. Entropy collapse as a blanket diagnosis misses that the *location* of exploration matters.
There's also a more deflationary view in the corpus: maybe RL isn't expanding reasoning capability at all, so 'scaling' is the wrong frame. The evidence that base models already contain reasoning in latent form — and that RL mostly learns *when* to deploy it rather than *how* to do it, recovering most gains by routing tokens — implies the ceiling may be a capability the base model already has, not something exploration unlocks Does RL post-training create reasoning or just deploy it?. If that's right, preserving entropy buys you better access to existing skills, not new ones.
And the corpus points to bottlenecks that have nothing to do with entropy. Numerical rewards plateau because they carry no information about *why* an answer failed; swapping in natural-language critiques breaks plateaus that more reward signal couldn't Can natural language feedback overcome numerical reward plateaus?. Binary correctness rewards quietly degrade calibration by rewarding confident wrong guesses, fixable only by adding a proper scoring rule Does binary reward training hurt model calibration?. And the reward signal's statistical structure itself — cross-rollout variance used to both weight tokens and filter degenerate queries — drives stability and speed independent of exploration Can one statistical measure serve dual purposes in RL training?.
The honest read: entropy collapse is the best-characterized and most clearly *predictive* bottleneck in this corpus, with a quantified ceiling and cross-domain replication. But calling it *the* main one overstates a moving target — the binding constraint shifts from execution to planning over training, may be capped by latent base-model capability, and competes with reward-information and calibration failures that entropy-preservation alone won't touch. The more defensible claim is that entropy collapse is the bottleneck that scaling *naively* (more steps, more reward) runs into first — which is exactly why it looks primary.
Sources 7 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.