Reinforcement training tends to sharpen what a model already knew rather than teach it new things — can any variant fix that?

This explores whether variants of on-policy RL can dodge probability squeezing — the tendency of reward maximization to compress a model's output distribution onto solutions it could already reach, rather than expanding what it can do. The honest starting point in the corpus is that the squeezing isn't an implementation bug you patch out; it's what on-policy reward maximization *does*. Does RLVR actually expand what models can reason about? shows via pass@k that base models actually beat their RLVR-trained versions at high k: the training raises sampling efficiency by concentrating probability on solutions already latent in the base distribution, but the reachable set doesn't grow. So 'squeezing' and 'the improvement' are often the same mechanism viewed from two angles.

That same compression shows up under different names across the corpus, which suggests it's structural. Does policy entropy collapse limit reasoning performance in RL? gives it an empirical law — performance saturates as policy entropy heads to zero — and Does reinforcement learning squeeze exploration diversity in search agents? finds the identical entropy-collapse signature in search agents, not just reasoning. Meanwhile Does binary reward training hurt model calibration? shows the reward *shape* itself pushes the model toward confident narrowing: binary correctness rewards never penalize confident wrong answers, so the policy learns to squeeze probability onto high-confidence guesses.

Here's the interesting part for your question: the corpus does describe on-policy interventions that manage the squeezing rather than abolish it. Entropy-collapse work names specific mechanisms — Clip-Cov, KL-Cov, GPPO — that deliberately preserve exploratory capacity by controlling *how fast* entropy drops during training Does policy entropy collapse limit reasoning performance in RL?. These are on-policy variants, and they buy you a higher performance ceiling by not letting the distribution collapse as hard. But note what they don't claim: they slow the squeeze, they don't reverse it into genuine capability expansion.

The more suggestive escape routes change what signal the policy optimizes on. Can natural language feedback overcome numerical reward plateaus? argues numerical rewards plateau precisely because a scalar carries no information about *why* a solution failed — feeding chain-of-thought critiques back in lets models produce solutions they were stuck on, breaking the plateau that pure reward-scaling can't. Can an agent's own beliefs guide credit assignment without critics? replaces sparse terminal reward with a dense per-turn belief-shift signal, and Can models improve themselves using only majority voting? bootstraps reward from consensus on unlabeled data. These are still on-policy, but they attack the information-starvation that makes squeezing so severe.

The thing you might not have expected: the corpus repeatedly pairs on-policy RL with SFT or distillation as the actual expander. In both the search-diversity and the RLVR-boundary findings, the move that *adds* new reachable behavior is off-policy — SFT on diverse demonstrations preserves exploration breadth Does reinforcement learning squeeze exploration diversity in search agents?, and distillation genuinely transfers reasoning patterns RL cannot Does RLVR actually expand what models can reason about?. So the corpus's answer is closer to: on-policy variants can *mitigate* squeezing (entropy control, richer feedback, denser reward), but the clean way to *avoid* the capability ceiling is to stop being purely on-policy — inject diverse off-policy data alongside it.