After AI is trained with reinforcement learning, has it learned new ways to reason, or just when to use skills it had?
Does RL teach models new reasoning or just better timing?
This explores whether reinforcement learning (RL) after pretraining gives language models genuinely new reasoning skills, or only teaches them when to switch on skills they already have.
This explores whether RL post-training gives language models genuinely new reasoning skills, or only teaches them when to use skills they picked up in pretraining. The corpus leans toward "timing" for ordinary reasoning tasks. It also holds a strong counter-case, so the honest answer is that it depends on what you train on and for how long.
The timing case is stronger than you might expect. Base models seem to already contain the machinery: the activation patterns for reasoning strategies exist before any RL happens. A hybrid that steered a base model using a thinking model's signals recovered 91% of the performance gains while using only 12% of the tokens (Does RL post-training create reasoning or just deploy it?, Does RL teach reasoning or just when to use it?). Give a base model enough attempts and it beats the RL-trained version, which suggests RL narrows sampling toward answers the base model could already reach. Distillation, by contrast, does transfer new reasoning patterns (Does RLVR actually expand what models can reason about?).
Two more findings fit this picture. One training example can be enough, and random-ish "spurious" rewards work almost as well as correct ones when the model is already well pretrained (What does reward learning actually do to model reasoning?, How does RL training reshape reasoning and what gets lost?). That is hard to square with RL as a teacher, and easy to square with RL as a trigger. RL also touches only 5–30% of parameters, and nearly the same ones across random seeds (Does reinforcement learning update only a small fraction of parameters?). That looks more like a targeted nudge than a rewrite.
The counter-case says the negative results may come from short training on narrow math. Prolonged RL with KL control, periodic policy resetting, and non-math tasks beat base models at every pass@k level, so the boundary of what the model can solve actually moved (Can reinforcement learning discover reasoning strategies base models cannot?). The split seems to follow task difficulty. On standard reasoning, RL activates latent ability. On complex multi-step planning, it found strategies the base model never produced, even with extensive sampling (Does reinforcement learning create new reasoning abilities or activate existing ones?). A controlled study adds the conditions: RL extends reasoning only when pretraining left headroom and the RL data targets tasks at the edge of the model's competence (When does RL actually extend reasoning beyond pretraining?).
Putting these together, pretraining supplies the building blocks and RL either sharpens their use or, under the right conditions, recombines them into something new. One hint about where novelty shows up: RL training tends to go through two phases. First it masters execution, and then planning becomes the bottleneck, with planning tokens getting more exploratory (Does RL training follow a predictable two-phase learning sequence?). Even "just timing" changes the model in other ways. Binary right/wrong rewards push models toward confident guessing and worse calibration, which a Brier-score term fixes (Does binary reward training hurt model calibration?). Training order also reshapes how exploratory the model stays across task types (Does training order reshape how models handle different task types?). If you want a model to solve problems it can't solve at all, plain RL on math probably won't do it. Distillation, or long, diverse, edge-of-competence RL, might.
Sources 12 notes
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
Pre-training acquires reasoning capability; RL teaches efficient deployment. A hybrid model combining base reasoning with thinking model steering recovered 91% of performance gains using only 12% of tokens, suggesting RL acts as a deployment optimizer rather than a capability creator.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Research shows that verifiable rewards act as catalysts that surface existing capabilities from pretraining, not teachers that build new reasoning. RL updates are structurally sparse and bounded by the pretrained prior, not algorithmic sophistication.
Show all 12 sources
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
RL-trained models outperform base models across all pass@k levels when trained with KL control, policy resetting, and non-mathematical tasks. This shows RL can expand capability boundaries, not just optimize sampling efficiency, especially on domains where base models lack established patterns.
For standard reasoning tasks, RL activates latent abilities already present in base models. For complex planning requiring multi-step coordination, RL generates genuinely novel strategies inaccessible to base models even with extensive sampling.
A controlled synthetic framework shows RL produces true capability gains only when pretraining established reasoning primitives and RL data targets tasks at the boundary of the model's competence. Without these conditions, RL refines sampling rather than extending capability.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Eliciting Reasoning in Language Models with Cognitive Tools
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- The Landscape of Agentic Reinforcement Learning for LLMs: A Survey
- From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains