Does extra training with rewards pay off less than just letting an AI think longer before answering?
Does RL scaling deliver much worse returns than inference scaling?
This explores whether spending compute on reinforcement learning (training a model further with rewards) gets you less improvement per unit of compute than spending it at answer time (letting the model think longer or try more), and what the collection says about how the two relate.
This explores whether RL training gives you less for your compute than inference-time scaling does. The collection doesn't contain a direct comparison of the two at equal cost. What it does show is more useful: the two aren't really rivals, because RL is often what makes extra thinking time worth paying for. Models trained to reason keep beating non-reasoning models however much inference compute the non-reasoning models get. Training gives a model a reasoning procedure that turns extra tokens into progress instead of filler Can non-reasoning models catch up with more compute?. So the question 'which scales better?' partly breaks down. Inference scaling pays off most on a model whose training has taught it how to use the extra time.
The case that inference scaling looks cheap comes from showing that a smaller model given more thinking time can match a much larger model on hard prompts Can inference compute replace scaling up model size?. Note what that result compares, though: inference compute against model size, not against RL. RL's own scaling has turned out to be orderly. A study using about 400K GPU-hours found that RL performance follows an S-shaped curve. Gains come fast at first and then level off at a ceiling, and the training recipe sets where that ceiling sits Does RL training follow predictable scaling curves?. That changes the question. If RL looks weak, you may be near the top of a curve whose ceiling a better recipe could raise. More compute on the same recipe won't.
Several notes help explain why RL's ceiling can be low. RL changes only 5–30% of a model's parameters, and it picks nearly the same subset every run Does reinforcement learning update only a small fraction of parameters?. Within the first pass over the training data, it tends to lock onto one style of answer the model already learned during pretraining and squeeze out the alternatives Does RL training collapse format diversity in pretrained models?. Both findings suggest RL mostly strengthens and selects habits the model already has rather than adding new ones, and that limits how far it can scale. Wasted training signal makes things worse. If every attempt at a prompt scores about the same, the model learns little from it and drifts toward generic template answers Why do language models collapse into generic templates?. If the problems are nearly impossible, the rare lucky success gets rewarded, which teaches shortcuts and can damage skills the model already had Do overly hard RLVR samples actually harm model capabilities?. Fixes that filter for informative prompts can make training 2–3× faster Can one statistical measure serve dual purposes in RL training?. So RL's poor returns are partly a data-selection problem, and you can work on that.
One caution applies to both approaches. On real constrained-optimization problems, models stall at about 55–60% of constraints satisfied, whatever their size or training, and reasoning-trained models don't clearly beat standard ones Do larger language models solve constrained optimization better?. Some limits hold no matter which kind of compute you add. There's also a third option. Looped architectures, which run the same layers several times, get reasoning gains from repeated computation rather than more parameters or more RL Can models learn by looping instead of growing larger?. Looped models trained to match written-out reasoning step by step can equal explicit chain-of-thought while cutting thinking time by 2.5–6.9× Can latent reasoning close the scaling gap with explicit chain-of-thought?.
The short answer: per unit of compute, RL often does look worse, but mostly for reasons you can change. Those are the recipe's ceiling, wasted training examples, and RL's habit of amplifying what pretraining already put there. Inference scaling looks cheap partly because RL already did the work that makes longer thinking pay off.
Sources 11 notes
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Snell et al. (2024) showed that inference-time compute trades off against model parameter scaling, especially on difficult prompts. This reveals pretraining and inference compute are not independent resources.
Large-scale study (400K GPU-hours, 200+ models) shows RL performance scales sigmoidally. Recipe choices set the ceiling; implementation details only affect efficiency. Stable recipes enable reliable extrapolation from small runs.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Show all 11 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Across constrained-optimization tasks, LLMs converge to ~55–60% constraint satisfaction independent of architecture, parameter count, or training regime. Reasoning models do not systematically outperform standard models, suggesting a fundamental ceiling rather than a scaling gap.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
LOTUS, a looped Transformer trained with gold CoT supervision at each latent position, closes the latent-to-explicit gap on GSM8K and cuts thought-phase latency by 2.5× to 6.9×. Latent reasoning need not be verbalized token-by-token to remain legible and competitive.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Sharpening Tax in Post-Training
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- Bridging the Gap Between Latent and Explicit Reasoning with Looped Transformers
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models