Extra thinking time only makes AI smarter if training first taught it how to use that time — which is exactly what reinforcement learning does.
What role does reinforcement learning play in optimizing inference compute?
This explores whether reinforcement learning actually *creates* the reasoning that makes extra inference compute pay off, or whether it just teaches models to spend that compute more efficiently on abilities they already had.
This explores whether reinforcement learning actually *creates* the reasoning that makes extra inference compute pay off, or whether it just teaches models to spend that compute more efficiently. The corpus points hard toward the second reading: RL's main job isn't to invent new reasoning, it's to make each token of inference compute productive. Reasoning models beat non-reasoning ones no matter how large the inference budget you hand the latter, because training installs a protocol that turns extra tokens into progress rather than wandering Can non-reasoning models catch up with more compute?. Inference compute is only as valuable as the training that taught the model what to do with it.
So what exactly does RL change? Several notes converge on a surprisingly narrow answer: it sharpens sampling rather than expanding capability. RLVR (RL from verifiable rewards) improves the odds of hitting a good solution the model could already produce, but pass@k analysis shows base models actually *overtake* RLVR-tuned ones at high sampling budgets — meaning RL narrows the distribution toward known-good answers rather than unlocking new ones Does RLVR actually expand what models can reason about?. It behaves like a catalyst that surfaces pretrained strategies, not a teacher of new skills, and remarkably a single training example — or even spurious rewards — can trigger the activation What does reward learning actually do to model reasoning? How does RL training reshape reasoning and what gets lost?. Structurally, RL touches only 5–30% of parameters in sparse-but-full-rank subnetworks, reinforcing the picture of RL as a targeted re-weighting of existing machinery rather than a rebuild Does reinforcement learning update only a small fraction of parameters?.
Where RL becomes genuinely a *compute-optimization* lever is in teaching models to spend inference adaptively. The naive approach is a fixed thinking budget for every prompt; the better one gives easy prompts less and hard prompts more, and this adaptive allocation beats simply using a bigger model under a uniform budget Can we allocate inference compute based on prompt difficulty?. RL is how a model learns that policy from experience: Thinkless uses a decoupled variant (DeGRPO) to train a single model to route between extended reasoning and fast direct answers, self-calibrating when to think without needing anyone to label which prompts are hard Can models learn when to think versus respond quickly?. That is inference-compute optimization in the most literal sense — RL deciding *how much* compute to burn per query.
The interesting frontier is what breaks RL out of the 'just re-sample the base model' ceiling. Two threads matter. First, richer reward signals: purely numerical rewards plateau because they say *that* you failed but not *why*, and swapping in natural-language critiques lets stuck models climb again Can natural language feedback overcome numerical reward plateaus?. Second, moving RL earlier — treating chain-of-thought as an exploratory action during pretraining, rewarded by information gain, plants the reasoning habits before RLVR ever runs, lifting benchmarks ~19% Can chain-of-thought reasoning be learned during pretraining itself?. And when the goal really is to install *new* knowledge rather than resurface old, reward-shaped generation (RLAG) that scores explanation quality outperforms plain supervised fine-tuning Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?.
The thing worth walking away with: 'optimizing inference compute' isn't one problem but two, and RL sits differently in each. For *efficiency* — spend the right amount per prompt — RL is the star, learning adaptive routing policies no fixed heuristic matches. But for raising the *ceiling* of what more compute can buy, RL as usually practiced mostly reshuffles the base model's existing distribution; the real gains come from architecture (looped/recurrent depth that adds effective compute without more parameters Can models learn by looping instead of growing larger?, or scaling laws tuned for inference throughput Can architecture choices improve inference efficiency without sacrificing accuracy?) and from where and how you inject the learning signal. RL optimizes how compute is *spent*, not, by itself, how much it's ultimately *worth*.
Sources 12 notes
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Research shows that verifiable rewards act as catalysts that surface existing capabilities from pretraining, not teachers that build new reasoning. RL updates are structurally sparse and bounded by the pretrained prior, not algorithmic sophistication.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Show all 12 sources
Research shows inference effectiveness varies dramatically by prompt difficulty. Reallocating the same total compute adaptively—giving easy prompts less and hard ones more—substantially outperforms larger models under uniform budgets.
Thinkless trains a single model to select between extended reasoning and direct responses using DeGRPO, which decouples mode selection from answer refinement. This prevents mode collapse and enables self-calibrated routing without explicit difficulty labels.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Models that re-apply layers in recurrent depth outperform larger feedforward networks on reasoning tasks. This works because recursion enables state tracking and compositional generalization that parameter scaling alone cannot achieve, with convergence signals providing natural halting.
Augmenting scaling laws with hidden size, MLP-to-attention ratio, and GQA configuration enables architecture optimization for inference. Optimized models achieved up to 2.1% higher accuracy and 42% greater throughput than LLaMA-3.2 under identical training budgets.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Eliciting Reasoning in Language Models with Cognitive Tools
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Spurious Rewards: Rethinking Training Signals in RLVR
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Rethinking Thinking Tokens: LLMs as Improvement Operators
- When More Thinking Hurts: Overthinking in LLM Test-Time Compute Scaling
- Base Models Know How to Reason, Thinking Models Learn When