When an AI is trained purely on rewards, it eventually becomes so certain of its approach that it stops discovering better ones.

This explores why reasoning models trained with pure RL keep improving for a while and then flatten out — and the corpus points to a single culprit with a surprisingly clean signature. As RL rewards the model for correct answers, the policy becomes more and more confident, its entropy falls, and exploration dries up. One line of work makes this almost quantitative: performance follows the law R = -a·exp(H) + b, so once entropy H approaches zero, reward saturates and no amount of extra compute buys more reasoning Does policy entropy collapse limit reasoning performance in RL?. Entropy collapse isn't a side effect of scaling — it *is* the ceiling.

The mechanism generalizes beyond math reasoning. The same convergence-onto-a-narrow-strategy shows up in search agents, where RL compresses behavioral diversity exactly as it does in reasoning, and where training on diverse demonstrations (SFT) is what preserves the breadth RL erodes Does reinforcement learning squeeze exploration diversity in search agents?. There's also a finer-grained account of *why* the policy narrows: when the reward has little variance within a prompt — every rollout scores about the same — the useful gradient vanishes and regularization quietly pushes the model toward generic, input-agnostic templates. Filtering for high-variance prompts before each update recovers the lost performance Why do language models collapse into generic templates?. So collapse can come both from over-confidence (entropy → 0) and from the reward signal going flat.

Why does this cap reasoning specifically? Because of what zero RL is actually doing. Evidence suggests RL post-training mostly teaches a base model *when* to deploy reasoning it already latently has, not *how* to reason in new ways — hybrid models recover ~91% of the gains just by routing tokens, and the activation directions for reasoning strategies exist before any RL Does RL post-training create reasoning or just deploy it?. If RL is selecting among pre-existing behaviors rather than inventing them, then collapsing the exploration distribution can only ever converge onto the best behavior the base model already contained — which is precisely a ceiling.

The interesting part is that the ceiling isn't uniform across training. RL reasoning tends to move through two phases: first the model nails execution correctness, then strategic planning becomes the bottleneck — and notably, planning-token entropy *rises* in the second phase even as execution entropy stabilizes Does RL training follow a predictable two-phase learning sequence?. That suggests entropy collapse is most damaging when it hits the tokens where exploration still matters (the planning decisions), which is why blanket entropy management can be blunt. It also hints that scale interacts with collapse: on social reasoning, larger models develop genuine belief-tracking under RL while smaller ones collapse into shortcut heuristics that hit the same accuracy without real reasoning Does reinforcement learning on theory of mind collapse with model scale?.

The corpus's escape routes all attack the information bottleneck rather than just clamping entropy. Targeted interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by controlling *how* entropy is spent Does policy entropy collapse limit reasoning performance in RL?. More radically, the ceiling may be a property of the reward *format*: scalar rewards carry no information about *why* an answer failed, and swapping in chain-of-thought critiques lets models generate correct solutions past a plateau where numerical rewards stall Can natural language feedback overcome numerical reward plateaus?. The takeaway a curious reader might not expect: entropy collapse isn't only a diversity problem you regularize away — it's a symptom that the reward signal has run out of things to teach, and the fix is often richer feedback, not more RL steps.