Can reinforcement learning add missing domain knowledge to fine-tuned reasoning models?
This explores whether reinforcement learning can teach a reasoning model facts it doesn't already know — or whether RL only reorganizes and surfaces knowledge that's already baked in from pretraining.
This explores whether reinforcement learning can add missing domain knowledge to a reasoning model, or whether it only sharpens access to knowledge the model already holds. The corpus is unusually opinionated here, and it pulls in two directions worth holding side by side. The dominant line of evidence says RL is an elicitation tool, not a knowledge-injection tool. RLVR doesn't push past the base model's reasoning boundaries — pass@k analysis shows base models actually match or beat RLVR-trained ones when you sample enough times, meaning RL narrows toward solutions already in the model's distribution rather than discovering new ones Does RLVR actually expand what models can reason about?. The same dynamic shows up elsewhere: a single training example can trigger the gains, and even spurious rewards work nearly as well as correct ones, which only makes sense if RL is activating pretrained strategies rather than teaching content What does reward learning actually do to model reasoning?. Several independent mechanisms — RL steering, decoding tweaks, feature steering — all unlock the same latent capability, suggesting post-training *selects* reasoning rather than creating it Do base models already contain hidden reasoning ability?, and one framing sharpens this to a slogan: RL teaches the model *when* to reason, not *how* Does RL post-training create reasoning or just deploy it?.
The most pointed answer to your literal question comes from medical-domain work showing RL improves domain reasoning by *pruning* wrong knowledge, not adding right knowledge — a +12.4 point gain achieved by suppressing low-reward trajectories that invoke incorrect facts Does RL improve domain reasoning by adding knowledge or removing it?. If the knowledge simply isn't there, this hits a wall that the corpus draws explicitly in a neighboring domain: prompt optimization can only reorganize what already exists and cannot compensate for missing foundational knowledge Can prompt optimization teach models knowledge they lack?. By that logic, plain RL inherits the same ceiling.
But the corpus also holds a genuine counterweight, and this is the part worth knowing. RL *from augmented generation* (RLAG) does appear to internalize domain knowledge — by cycling between augmented and unaugmented generation and rewarding explanation quality, not just answer correctness, it embeds knowledge structures more effectively than supervised fine-tuning Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. The trick is that the missing knowledge is supplied through augmentation during training; RL is the mechanism that internalizes it coherently. In the same spirit, other work argues sophisticated domain reasoning can *emerge* from simple accuracy rewards on hard problems without teacher distillation Can simple rewards alone teach complex domain reasoning?.
So the honest synthesis: vanilla reward-based RL almost certainly can't conjure domain knowledge that pretraining never saw — it elicits, prunes, and re-routes. What *can* add genuinely new content is the channel feeding the RL loop. Natural-language critique feedback breaks plateaus that numerical rewards can't, precisely because the critique carries information about *why* an answer failed that a scalar reward lacks Can natural language feedback overcome numerical reward plateaus?, and reframing where reasoning gets planted — treating chain-of-thought as an exploratory action rewarded during *pretraining* — lifts capability earlier in the pipeline rather than bolting it on after Can chain-of-thought reasoning be learned during pretraining itself?.
The thing you may not have known you wanted: there's a quiet warning in the corpus about *fine-tuned* reasoning models specifically. Fine-tuning can make reasoning chains *performative* — the steps stop causally driving the answer, so a model that looks like it's reasoning over domain facts may not actually be using them Does fine-tuning disconnect reasoning steps from final answers?. That reframes your question: before asking whether RL can add missing knowledge, it's worth asking whether the fine-tuned model's reasoning is even wired to the knowledge it already has.
Sources 11 notes
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
RL enhances medical reasoning by suppressing incorrect domain knowledge during reasoning—not by expanding what models know. Evidence shows RL achieves +12.4 point knowledge improvement by removing low-reward reasoning trajectories that invoke wrong facts.
Prompting works entirely within a model's pre-existing training distribution and cannot supply domain knowledge absent from training data. This creates a hard ceiling: no prompt strategy can compensate for missing foundational knowledge, only reorganize what already exists.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Medical AI systems and o3 demonstrate that sophisticated domain reasoning emerges naturally from RL training on difficult problems with only basic accuracy signals, without requiring explicit chain-of-thought distillation from teacher models.
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
RLP treats CoT as exploratory action during pretraining, using log-likelihood improvement as verifier-free reward. Applied to Qwen3-1.7B and Nemotron-Nano-12B, the method improves math and science benchmarks substantially, suggesting reasoning can be planted earlier in training.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.