Can AI training with rewards and feedback only work on a model that already learned enough beforehand?
Does reinforcement learning require sufficient pretraining to be effective?
This explores whether RL post-training only works once a model has learned enough in pretraining, and what 'enough' means. The corpus says mostly yes, with a twist: pretraining has to leave the right kind of gap for RL to fill, and it also has to be big enough.
This explores whether RL post-training only works once a model has learned enough in pretraining, and what 'enough' means. The corpus says mostly yes, with a twist: pretraining has to leave the right kind of gap for RL to fill, and it also has to be big enough.
The clearest evidence is a controlled synthetic study. RL produces real new reasoning ability only when two things hold. Pretraining must already have planted the basic reasoning building blocks, and the RL problems must sit right at the edge of what the model can currently do When does RL actually extend reasoning beyond pretraining?. If either is missing, RL mostly reshuffles how the model samples answers it could already give, and it adds no new capability. A chess testbed puts numbers on the dependency. Across models from 5M to 1B parameters, pretraining loss predicts how well a model does after RL. How fast RL improves also scales roughly linearly with the tokens spent on pretraining Does pretraining loss predict performance after reinforcement learning?. So pretraining sets both the ceiling and the rate. The authors add that as total budgets grow, the best split of compute shifts toward RL, which means RL adds to pretraining and does not replace it.
RL training itself shows why this dependence exists. Across eight models, RL first consolidates execution, getting the steps right, and only then does strategic planning become the bottleneck Does RL training follow a predictable two-phase learning sequence?. That first phase only makes sense if the model can already attempt the steps. Two cheaper tricks lean on existing ability in a similar way, though this part is my reading. Training on negative samples alone matches full RL by suppressing wrong trajectories while keeping diversity Does negative reinforcement alone outperform full reinforcement learning?, which prunes a distribution that already contains good answers. Test-time RL with majority-vote rewards works because consensus answers tend to be correct Can models improve themselves using only majority voting?, and that only holds if the base model is right often enough.
The line between pretraining and RL is also blurring. One approach treats chain-of-thought as an exploratory action during pretraining, rewarded by how much it improves prediction, and lifts reasoning by about 19 percent Can chain-of-thought reasoning be learned during pretraining itself?. Another runs RL on Wikipedia during pretraining and actively picks content the model hasn't mastered yet Can reinforcement learning improve models during general pretraining?. A third recasts next-token prediction as a reasoning task with verifiable rewards taken from the corpus itself Can next-token prediction become a reasoning task with RL?. That changes the question from whether RL needs pretraining to whether RL can be part of pretraining. Picking not-yet-mastered content is the same edge-of-competence principle as in the synthetic study, applied earlier in training.
There is also a middle stage. Agents that learn from the consequences of their own actions, with no external reward, make a better warm start for later RL than expert imitation alone Can agents learn from their own actions without external rewards?. That is a bridge that prepares a model for RL. The dramatic 'reasoning emerges from simple rewards' results Can simple rewards alone teach complex domain reasoning? show how powerful RL can be, but they start from strong pretrained models. The corpus has no clean test of RL from a blank slate on real language tasks. Its tightest evidence comes from controlled synthetic and chess settings, so the general claim rests on those.
Sources 10 notes
A controlled synthetic framework shows RL produces true capability gains only when pretraining established reasoning primitives and RL data targets tasks at the boundary of the model's competence. Without these conditions, RL refines sampling rather than extending capability.
In a controlled chess testbed, models pretrained from 5M to 1B parameters show a joint scaling law where pretraining loss predicts final RL performance, and RL improvement rate scales roughly linearly with pretraining tokens. This suggests optimal compute allocation shifts toward RL as total budgets grow.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Show all 10 sources
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
PretrainZero shows that RL during pretraining on Wikipedia, combined with active selection of not-yet-mastered content, outperforms standard pretraining and random reinforcement. The gain comes from *which* content is reinforced, not new data.
Reinforcement Pre-Training transforms next-token prediction into a reasoning task by providing verifiable rewards from the corpus itself, eliminating reward hacking and enabling inference-time scaling during pretraining. This suggests token-level reasoning patterns during pretraining strengthen downstream RL fine-tuning.
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
Medical AI systems and o3 demonstrate that sophisticated domain reasoning emerges naturally from RL training on difficult problems with only basic accuracy signals, without requiring explicit chain-of-thought distillation from teacher models.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Eliciting Reasoning in Language Models with Cognitive Tools
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- RLP: Reinforcement as a Pretraining Objective
- Understanding Reasoning from Pretraining to Post-Training
- Base Models Know How to Reason, Thinking Models Learn When
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining