Can an AI get good at chess just by studying past games, or does it need to practice and get rewarded?
Can pretraining alone achieve chess performance without reinforcement learning?
This explores whether a model that only learns from existing data (pretraining) can get good at chess, or whether it also needs reinforcement learning (RL), which trains by trial and reward.
This explores whether a model that only learns from existing data (pretraining) can get good at chess, or whether it also needs reinforcement learning (RL), which trains by trial and reward. The corpus has no clean pretraining-only versus RL showdown. Its one chess study shows the two are tightly coupled. Does pretraining loss predict performance after reinforcement learning? trained chess models from 5M to 1B parameters and found that pretraining loss predicts how well a model ends up after RL. It also found that the speed of RL improvement grows roughly linearly with the amount of pretraining data.
So pretraining sets the starting line and the slope, and RL does the climbing. A better-pretrained model gets more out of each unit of RL. The study also finds that as total compute budgets grow, the best split shifts toward RL. The study never stops training after pretraining, so it can't say where pretraining alone tops out. But if RL's payoff keeps growing with budget, a pretraining-only run at large scale probably leaves performance unclaimed.
Neighboring notes suggest why RL adds something pretraining doesn't. RL barely rewrites the model: it updates only 5 to 30 percent of parameters, and the same subnetwork shows up across random seeds (Does reinforcement learning update only a small fraction of parameters?). It also tends to amplify one pattern already latent from pretraining while suppressing the alternatives (Does RL training collapse format diversity in pretrained models?). Pretraining supplies the raw material, and RL selects and sharpens it. In reasoning tasks, much of the gain comes from pushing down bad attempts. Training on incorrect samples alone often matches full RL (Does negative reinforcement alone outperform full reinforcement learning?). That is the kind of correction pretraining on finished data can't easily supply.
The boundary between the two stages is also blurring, so the question may be more of a pipeline than a binary. One method puts an RL-style reward inside pretraining, scoring chain-of-thought by how much it improves next-token prediction, and lifts reasoning by 19 percent (Can chain-of-thought reasoning be learned during pretraining itself?). Another lets agents treat the consequences of their own actions as the teaching signal, with no external reward. It matches expert-dependent baselines with half the data and gives RL a better warm start (Can agents learn from their own actions without external rewards?). Tree search can also generate step-by-step quality signals with no human labels (Can tree search replace human feedback in LLM training?). In each case, feedback shows up earlier in training rather than being reserved for a final RL phase.
Sources 7 notes
In a controlled chess testbed, models pretrained from 5M to 1B parameters show a joint scaling law where pretraining loss predicts final RL performance, and RL improvement rate scales roughly linearly with pretraining tokens. This suggests optimal compute allocation shifts toward RL as total budgets grow.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
Show all 7 sources
Research across eight environments shows that agents can use future states from their own actions as supervision without external rewards, matching expert-dependent baselines with half the data and providing superior warm-starts for subsequent RL training.
AlphaLLM uses tree search outcomes and three critic models to derive dense reward signals equivalent to human-labeled feedback. Tree structure naturally ranks solution paths by success, replacing the annotation oracle that standard RLHF requires.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Understanding Reasoning from Pretraining to Post-Training
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- RLP: Reinforcement as a Pretraining Objective
- Base Models Know How to Reason, Thinking Models Learn When