Why does gradient discarding limit standard policy clipping?
This explores why the standard PPO/GRPO move of clipping — throwing away the gradient on any token whose probability ratio drifts too far — quietly caps what RL can teach a model, and what the corpus offers as alternatives.
This explores why the standard trick of *clipping* in policy-gradient RL — discarding the gradient whenever a token's update would push its probability ratio outside a trust region — ends up limiting learning rather than just stabilizing it. The intuition the corpus keeps circling back to: the tokens clipping throws away are not random. They are disproportionately the high-leverage, exploratory ones, so discarding their gradients systematically narrows what the model can become.
The sharpest evidence is the finding that policy entropy collapse is the *primary* bottleneck in RL for reasoning Does policy entropy collapse limit reasoning performance in RL?. There, performance follows a clean law and saturates as entropy drains toward zero. The interventions that work — Clip-Cov, KL-Cov, GPPO — are precisely the ones that stop blindly discarding gradients on high-covariance tokens and instead preserve a portion of that exploratory signal. In other words, standard clipping's gradient discarding *is* one of the engines of entropy collapse: it keeps amputating the updates that would have kept the policy curious. A related failure shows up when RL converges hard onto a single dominant pretraining format within the first epoch Does RL training collapse format diversity in pretrained models? — once the surviving gradients all point the same way, alternatives are suppressed rather than explored.
The deeper problem is that the scalar advantage clipping operates on is information-poor to begin with, so discarding any of it hurts more than it should. Numerical rewards carry no account of *why* a trajectory failed, which is why models stall on plateaus that natural-language critiques can break Can natural language feedback overcome numerical reward plateaus?. When the only signal is a thin scalar and clipping then zeroes out part of even that, there's little left to learn from. Approaches that convert rich, tokenized environment feedback into dense per-token credit assignment Can environment feedback replace scalar rewards in policy learning? are attacking the same wound from the other side: instead of accepting a sparse signal and clipping it further, they manufacture more gradient where standard methods have none.
Discarding also interacts badly with how advantages get normalized. With overly hard samples, group-relative normalization treats a rare accidental success as a high-advantage trajectory and reinforces it — amplifying shortcuts and answer-repetition instead of reasoning Do overly hard RLVR samples actually harm model capabilities?. Clipping doesn't fix this; it just decides *which* of these distorted gradients survive. The constructive alternatives in the corpus tend to be about shaping the signal before it ever reaches the clip: giving partial solution traces on hard problems so the gradient is informative rather than sparse Can adaptive guidance from solution traces reduce reward sparsity in RL?, or processing successful and failed episodes asymmetrically so failures still teach something Should successful and failed episodes be processed differently?.
The thing worth taking away: clipping was designed as a *stability* mechanism, but the corpus reframes it as a *capacity* mechanism in disguise. Because RL already touches only a small, structured slice of parameters Does reinforcement learning update only a small fraction of parameters?, every gradient that clipping discards is one fewer chance to move that narrow subnetwork in a useful direction — which is why the frontier of recent work is less about clipping better and more about not throwing the informative gradients away in the first place.
Sources 8 notes
Empirical law R = -a·exp(H) + b shows performance saturates when policy entropy approaches zero. Interventions like Clip-Cov, KL-Cov, and GPPO preserve exploratory capacity by managing entropy reduction during training.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
SDPO converts tokenized environment feedback into dense gradient signals by using the feedback-conditioned policy as a self-teacher. The policy, when given retrospective evidence of its mistakes in-context, implicitly acts as its own process reward model, making external reward signals unnecessary.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
GHPO dynamically provides ground-truth solution traces for hard problems while using standard RL for manageable ones, achieving 5% gains across math benchmarks. This converts wasted compute on impossible problems into learning signal by leveraging traces already present in training data.
SkillRL demonstrates that treating successful episodes as concrete demonstrations and failures as abstracted lessons achieves state-of-the-art performance on complex tasks while using substantially less context than uniform approaches. The asymmetry mirrors human expert reasoning and avoids the degradation seen in uniform consolidation methods.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.