When an AI gets better at a task after training, is it learning something new — or just learning how to phrase what it already knew?
Does RL primarily teach output format rather than new knowledge?
This explores whether reinforcement learning after pretraining mostly teaches a model how to shape and present its answers (format, structure, when to reason) rather than adding knowledge or reasoning ability it didn't have before.
This explores whether RL post-training mostly teaches models how to package what they already know, rather than teaching them anything new. Much of the corpus points toward "mostly yes," but with a twist. The format question first came up with instruction tuning, not RL. Models trained on empty or deliberately wrong instructions score about as well as models trained on correct ones, which suggests what carries over is a sense of what the output should look like, not an understanding of the task Does instruction tuning teach task understanding or output format?. RL seems to repeat this pattern. A 1.5B model given only lightweight LoRA updates matched much larger fully RL-trained models on reasoning benchmarks. That's hard to explain if RL were writing new knowledge into the model, and easy to explain if it mainly reorganizes how answers are laid out Can small models reason well by just learning output format?.
The stronger claim is that RL teaches *when* to reason, not *how*. Reasoning strategies show up as activation patterns in base models before any RL happens. Hybrid models recover about 91% of RL's gains just by deciding which tokens get the 'thinking' treatment Does RL post-training create reasoning or just deploy it?. Tests that change a familiar problem slightly point the same way: GRPO-trained models drop sharply on these near-variants, which looks more like sharper template-matching than a learned procedure Do fine-tuned language models actually learn optimization procedures?.
The surprising part is that RL doesn't just teach *a* format. It picks one. In controlled experiments, RL boosts a single output style that was already present in pretraining and suppresses the others, usually within the first epoch. Which style wins depends on model size more than on which style works best Does RL training collapse format diversity in pretrained models?. Badly tuned RL can make this narrowing worse. When rewards barely differ between attempts on the same prompt, the model drifts toward generic templates that ignore the input Why do language models collapse into generic templates?. Problems that are too hard reward lucky shortcuts, such as repeating answers or skipping computation, and those habits damage skills the model already had Do overly hard RLVR samples actually harm model capabilities?.
The counter-evidence is real, though. With longer training, diverse non-math tasks, KL control (a penalty that keeps the model from drifting too far from where it started), and periodic policy resets, RL-trained models beat their base models at every pass@k level. That means they solve problems the base model couldn't solve even with many tries, which is the usual test for 'new capability' rather than 'better sampling' Can reinforcement learning discover reasoning strategies base models cannot?. Results in long, multi-step software tasks Can reinforcement learning scale beyond single-turn language tasks? and tool-using math agents trained on carefully filtered trajectories Why do correct code trajectories teach models to tolerate errors? also suggest RL can build working skills, at least in environments with rich feedback.
Put together, the corpus suggests 'format vs. knowledge' is the wrong split. In its default short-run setup, RL mostly selects and sharpens what pretraining already put there. Whether it adds more depends on the training recipe: diverse tasks, rewards that actually vary, sensible difficulty, and enough time. One practical result is that RL works best when something else lays the groundwork first. Imitation-style training followed by verifiable-reward RL beats either one alone, because RL can only sharpen attempts that are already roughly right Does sequencing imitation then exploration training improve reasoning?.
Sources 11 notes
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
A 1.5B parameter model with LoRA-only post-training matched larger full-parameter RL models on reasoning tasks, suggesting RL teaches output format organization rather than new factual knowledge. This efficiency indicates reasoning and knowledge storage are separable capabilities.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Show all 11 sources
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
RL-trained models outperform base models across all pass@k levels when trained with KL control, policy resetting, and non-mathematical tasks. This shows RL can expand capability boundaries, not just optimize sampling efficiency, especially on domains where base models lack established patterns.
Modified DAPO training doubled SWE-bench Verified performance from 20% to 39% on Qwen2.5-72B, matching larger models. This demonstrates RL works in stateful multi-step environments with delayed rewards and complex feedback, beyond theoretical single-turn MDPs.
GRPO-RoC filters positive trajectories for quality while preserving diverse failures as negative signal, allowing a 14B model to reach frontier math performance in 510 RL steps, surpassing much larger models with cleaner reasoning.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- Sharpening Tax in Post-Training
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Eliciting Reasoning in Language Models with Cognitive Tools
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models