When AI budgets grow, should the extra money go into trial-and-error training, or into bigger models and more thinking time?
Should larger compute budgets allocate more resources to reinforcement learning?
This explores whether, as compute budgets grow, the extra should go to reinforcement learning (RL) training or to other uses such as bigger models, supervised fine-tuning, distillation or inference-time compute.
This explores whether, as compute budgets grow, the extra should go to reinforcement learning (RL) training or to other uses such as bigger models, supervised fine-tuning, distillation or inference-time compute. The corpus has no head-to-head study of budget splits, so this is what the pieces suggest together: RL is worth having, but more RL on its own looks like a diminishing-returns bet.
The case for RL is that it installs something extra compute can't buy later. Non-reasoning models don't catch up to reasoning models however many tokens they get at inference, because training instills a reasoning protocol that makes additional tokens productive Can non-reasoning models catch up with more compute?. RL can also beat supervised fine-tuning at specific jobs. For embedding domain knowledge, it rewards the quality of the explanation instead of token-by-token matching Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. So some RL looks close to a prerequisite.
The case against putting more into it is that RL mostly sharpens what a model already has. When you sample many attempts, base models beat RLVR-trained models (RL with automatically checkable rewards). RLVR narrows sampling toward answers the base model could already reach, while distillation is what transfers new reasoning patterns Does RLVR actually expand what models can reason about?. Search agents show the same squeeze. RL collapses their behavior onto a few reward-maximizing strategies, while SFT on diverse demonstrations keeps exploration broad Does reinforcement learning squeeze exploration diversity in search agents?. If the base model sets the ceiling, more RL compute gets you to it faster, not past it. RL also updates only 5–30% of parameters, in nearly identical subnetworks across random seeds Does reinforcement learning update only a small fraction of parameters?. Reading that as "RL is a small, structured nudge rather than a sponge for unlimited compute" is my inference, not the paper's claim.
How the RL compute is spent seems to matter more than how much. RL training moves through two phases: first execution correctness, then strategic planning as the bottleneck. Concentrating optimization on planning tokens produced the significant gains Does RL training follow a predictable two-phase learning sequence?. And when RL stalls on a plateau, numerical rewards don't say why an attempt failed. Chain-of-thought critiques let stuck models find correct solutions that scalar rewards never surfaced Can natural language feedback overcome numerical reward plateaus?. A bigger budget goes further if it buys richer feedback and targeted planning than if it buys more of the same rollouts.
Other uses also compete for the marginal compute. Giving easy prompts less inference compute and hard ones more beats uniform budgets, and even beats larger models under uniform budgets Can we allocate inference compute based on prompt difficulty?. Looped models get reasoning gains by re-applying layers instead of growing Can models learn by looping instead of growing larger?. Extra compute probably shouldn't default to "more RL." RL is a component to fund until sampling is efficient, and past that point the marginal budget looks better spent on diversity-preserving training, distillation, richer feedback or adaptive inference.
Sources 9 notes
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Show all 9 sources
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Efficient Reinforcement Learning via Large Language Model-based Search
- Reasoning Models Can Be Effective Without Thinking
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- RAGEN-2: Reasoning Collapse in Agentic RL