Does a bigger AI model or a better-trained one give reinforcement learning more to build on?

This explores whether a bigger or a better-composed pretrained model gives RL more to work with. The corpus suggests the two behave differently. Quantity gives a smooth, predictable payoff. Quality, meaning what kinds of behavior pretraining put in the model, decides what RL can reach at all. No note here varies data quality while holding token count fixed, so the picture is assembled from adjacent experiments.

The clearest evidence on quantity is a controlled chess study. Across models from 5M to 1B parameters, Does pretraining loss predict performance after reinforcement learning? finds that pretraining loss predicts how well the model does after RL. The rate at which RL improves the model also grows roughly linearly with the number of pretraining tokens. More pretraining doesn't just raise the starting line. It steepens the climb. The authors read this as a reason to shift compute toward RL as total budgets grow. It's a chess testbed, so treat the exact numbers as a demonstration of the pattern rather than a rule for language models.

Quality shows up as which behaviors were already in the model. In Why do random rewards improve reasoning for some models but not others?, random or even incorrect rewards lifted Qwen2.5-Math by 16–25% on MATH-500. The reward carried almost no information. It just activated code-style reasoning the model had absorbed in pretraining. Llama and OLMo, given the same treatment, gained nothing. The same RL recipe gave opposite results depending on what pretraining had installed. Does RL training collapse format diversity in pretrained models? adds a related twist. RL quickly amplifies one format from the pretraining mix and suppresses the others within the first epoch. Which format wins depends on model scale, not necessarily on which one performs best. Pretraining diversity is therefore something RL spends down, and it may not get it back.

This fits a general split of labor. Do pretraining and fine-tuning scale independently in language models? shows that scaling pretraining mostly improves stored knowledge, in lower layers, while scaling fine-tuning improves how that knowledge is expressed as behavior, in upper layers. That result is about fine-tuning, not RL specifically, but it points the same way. Post-training shapes and selects what pretraining supplies. It doesn't manufacture it. Do overly hard RLVR samples actually harm model capabilities? is the cautionary case. When problems sit beyond what the base model can do, RL doesn't teach real reasoning. It rewards lucky shortcuts like repeating an answer or skipping computation.

The practical takeaway is that more pretraining tokens raise the ceiling and speed of RL gains in a fairly predictable way, but composition decides whether those gains are reachable. A model whose pretraining never encoded the right behavior gives RL nothing to amplify, and one whose mix RL collapses onto a single format may lose flexibility it can't recover. What's missing from the corpus is a direct test that holds tokens fixed and varies data quality, and that's the experiment that would settle the question.