Why does reinforcement learning teach AI models so much less per example than simply showing them the right answer?
Do policy gradient methods carry less training signal than other RL algorithms?
This explores whether the policy-gradient style of reinforcement learning used to train today's LLMs (methods like PPO and GRPO) learns less from each training example than other approaches. The corpus doesn't really compare policy gradients against other RL families. What it does show is how little information RL carries compared with supervised fine-tuning, and how much of that small amount gets wasted.
This explores whether the policy-gradient methods used to train today's LLMs learn less per example than other approaches. The corpus doesn't line policy gradients up against other RL algorithms such as value-based methods. Its real comparison is between RL and supervised fine-tuning (SFT), and there the answer is a clear yes. When a model gets one reward for a whole attempt (right or wrong), it receives about one bit of information per episode. SFT gets a signal for every token it imitates. Thinking Machines Lab put a number on this: RL takes in roughly 1000 times less information per token than SFT. That explains a surprising result, which is that LoRA at rank 1 (one of the smallest possible adapters) matches full fine-tuning in RL Why does LoRA rank 1 work for reinforcement learning?. If so little information comes in, a tiny adapter has room to store all of it.
The model's weights show the same thinness. Across seven RL algorithms and ten model families, RL changed only 5–30% of parameters, with no regularization forcing it to. Those changes landed in nearly the same places across random seeds Does reinforcement learning update only a small fraction of parameters?. So the limited signal isn't spread randomly. It flows into a consistent, structured subnetwork. That fits other findings that RL mostly picks and strengthens something already in the model rather than teaching new skills. For example, it quickly amplifies one output format from pretraining and suppresses the others Does RL training collapse format diversity in pretrained models?.
The less obvious point is that the small signal RL does have is often wasted, and that is a bigger lever than the choice of algorithm. Group-based policy gradients learn by comparing several attempts at the same prompt. If every attempt succeeds or every attempt fails, there's nothing to compare, and the learning signal disappears. When reward variance stays low, regularization takes over and the model drifts toward generic templates that ignore the input. Filtering for prompts where outcomes vary restores performance Why do language models collapse into generic templates?. Problems that are nearly impossible are worse than useless. A rare lucky success gets treated as a high-value example, and the model learns shortcuts like repeating answers Do overly hard RLVR samples actually harm model capabilities?. Several fixes recover signal from this dead space. One gives partial solution hints on hard problems so they become learnable Can adaptive guidance from solution traces reduce reward sparsity in RL?. Another reuses the variation between attempts both to weight individual tokens and to filter out prompts that can't teach anything Can one statistical measure serve dual purposes in RL training?. A third concentrates updates on planning tokens, where most of the learning happens later in training Does RL training follow a predictable two-phase learning sequence?.
What the signal says matters too, not just how much there is. Training only on failures, by pushing down wrong answers, matches or beats full PPO and GRPO. Rewarding successes tends to pile probability onto a few answers and reduce diversity Does negative reinforcement alone outperform full reinforcement learning?. That narrowing shows up in reasoning models Why does RLVR training narrow a model's problem solving ability? and in search agents Does reinforcement learning squeeze exploration diversity in search agents?. Even a plain right/wrong reward hides information. It doesn't penalize confident wrong answers, so models learn to guess boldly. Adding a calibration term (the Brier score) puts that missing information back Does binary reward training hurt model calibration?.
The takeaway is that RL's thin signal is a property of sparse, outcome-only rewards, not of policy gradients specifically. That thinness is also why RL tends to refine abilities the model already has rather than add new ones. The most useful work on this lies less in swapping algorithms and more in designing rewards and choosing prompts so that each of RL's few bits carries real information.
Sources 12 notes
Thinking Machines Lab found LoRA at rank 1 achieves parity with full fine-tuning in RL, attributed to RL's O(1) bits of signal per episode versus supervised learning's O(tokens) bits. This means RL absorbs roughly 1000 times less information per token than SFT.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Show all 12 sources
GHPO dynamically provides ground-truth solution traces for hard problems while using standard RL for manageable ones, achieving 5% gains across math benchmarks. This converts wasted compute on impossible problems into learning signal by leveraging traces already present in training data.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
RLVR narrows models' problem-solving scope by prioritizing exploitation over exploration, a phenomenon called capability boundary collapse. Multiple importance sampling with exploration-based advantage functions can counteract this by integrating external data and explicitly rewarding discovery of underexplored but valuable reasoning paths.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Sharpening Tax in Post-Training
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- RAGEN-2: Reasoning Collapse in Agentic RL
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR