Understanding Reasoning from Pretraining to Post-Training

Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the pretraining that precedes it. As a result, two basic questions remain open: (1) how do pretraining choices (model size, data) shape the returns to RL compute, and (2) what does RL actually do to the model? These questions are difficult to study in the standard LLM setting: pretraining corpora are vast and uncontrolled, making it hard to attribute behaviors to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive. To address these challenges, we use chess as a controlled testbed for studying reasoning across the full pretraining-to-post-training pipeline. We follow the standard LLM training pipeline by pretraining language models from 5M to 1B parameters on human chess games, supervised fine-tuning on synthetic reasoning traces, and running RL on chess puzzles with verifiable rewards. Using this framework, we establish a scaling law connecting pretraining and RL: the post-RL performance at given RL compute level is well-predicted from the pretraining loss, and slope of the RL reward curves improves approximately linearly with the pretraining tokens.

Introduction. The standard pipeline for training large language models (LLMs) consists of large-scale pretraining followed by post-training, typically supervised fine-tuning (SFT) and reinforcement learning (RL) with verifiable rewards (Guo et al., 2025; Lambert et al., 2024; Yu et al., 2025; Zeng et al., 2025). As LLMs continue to scale, two perspectives on where to invest additional compute have begun to diverge. One emphasizes the pretrained prior: scaling model size, data, and compute to produce stronger base models from human text (Kaplan et al., 2020; Hoffmann et al., 2022). The other emphasizes experience: using RL to learn from environmental interaction and outcome-based feedback, thereby eliciting or developing capabilities beyond direct imitation. This view is reflected in arguments for an era of experience (Silver and Sutton, 2025), and exemplified by AlphaZero, which famously removed the imitation cold-start used by earlier AlphaGo and learned stronger policies from self-play alone (Silver et al., 2016, 2017).

Discussion / Conclusion. We used chess as a testbed for studying how pretraining influences RL dynamics and how RL reshapes the inherited policy. We established a joint scaling law: pretraining loss predicts post-RL performance level, while pretraining data scale is closely associated with the slope of RL improvement. The resulting compute-allocation frontier suggests a tradeoff between pretraining and RL. As the total budget grows, the optimal pretraining fraction tends to decrease, indicating that RL should take an increasingly large share of compute. Meanwhile, starting RL too early from weakly pretrained checkpoints gives limited gains in our setting, suggesting that RL remains initialization-dependent and requires sufficient pretraining exposure before it becomes effective. The behavior also differs between pass@1 and pass@16.