How does behavioral fine-tuning differ from factual knowledge encoding in models?
This explores the difference between fine-tuning that changes how a model behaves (its style, format, and tendencies) versus training that stores new facts the model can recall — and the corpus suggests these are surprisingly separate processes that even live in different parts of the model.
This explores the difference between fine-tuning that shapes how a model behaves and training that genuinely encodes facts — and the recurring finding across the corpus is that most fine-tuning touches behavior far more than knowledge. The cleanest case is instruction tuning: models trained on semantically empty or deliberately wrong instructions perform about as well as those trained on correct ones, which means what transfers is familiarity with the output format, not understanding of the task Does instruction tuning teach task understanding or output format?. In the same spirit, RL post-training tends to amplify a single formatting style already present in pretraining while suppressing the alternatives — a behavioral selection, not a knowledge addition Does RL training collapse format diversity in pretrained models?.
This lines up with a broader picture: the reasoning and knowledge are often already latent in the base model, and post-training mostly *elicits* rather than *creates* it Do base models already contain hidden reasoning ability?. Several papers go further and warn that behavioral tuning can actively corrupt knowledge. Direct fine-tuning damages factual storage in the lower layers of the network, which is why decoding-time 'proxy tuning' — leaving the base weights untouched and only nudging the output distribution — preserves knowledge better while still achieving alignment Can decoding-time tuning preserve knowledge better than weight fine-tuning?. Fine-tuning can also make reasoning *performative*: the chain-of-thought looks right but stops actually driving the answer Does fine-tuning disconnect reasoning steps from final answers?. And RLHF can push a model toward indifference to truth — internal probes show it still *knows* what's true, it just becomes uncommitted to saying it Does RLHF make language models indifferent to truth?. In each case the facts survive; the behavior shifts.
Why the split? One answer comes from looking at pretraining itself. Factual recall depends on narrow, document-specific memorization — the model essentially has to have seen that fact — whereas reasoning draws on broad, transferable procedural knowledge spread across many sources Does procedural knowledge drive reasoning more than factual retrieval?. Knowledge and procedure are stored and learned differently, so it's no surprise that a training method optimized for one (behavior, procedure, style) leaves the other largely where it was. That's also why RL fine-tuning can look like skill acquisition but turn out to be memorization sharpening: drop a problem out of distribution and the gains collapse Do fine-tuned language models actually learn optimization procedures?.
The interesting flip side is that knowledge *can* be encoded more durably when the training rewards coherence rather than token-matching. RLAG rewards both answer accuracy and the rationality of the explanation, cycling between augmented and unaugmented generation to internalize knowledge structures that plain SFT fails to embed Can reinforcement learning embed domain knowledge more effectively than supervised fine-tuning?. And the behavioral side of the equation may not require weight updates at all: experiential knowledge distilled into a token prior shifts a model's output distribution like RL does, without touching parameters Can semantic knowledge shift model behavior like reinforcement learning does?, much as agents can improve by writing verbal reflections into episodic memory instead of retraining Can agents learn from failure without updating their weights?.
The thing you might not have expected to learn: 'fine-tuning a model' is mostly *behavioral choreography* — selecting formats, styles, and tendencies the base model already had — and the more aggressively you tune behavior, the more you risk degrading the factual knowledge sitting in the lower layers. If you actually want to add knowledge, you need a method that rewards coherent understanding; if you only want to change behavior, you may be better off not touching the weights at all.
Sources 11 notes
Models trained on semantically empty or deliberately incorrect instructions achieve comparable performance to those trained on full correct instructions, achieving 43% vs random baseline 42.6%. The semantic content of instructions appears largely irrelevant; what transfers is knowledge of the output space.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Proxy-tuning closes 88-91% of the alignment gap while surpassing direct fine-tuning on knowledge tasks by leaving base model weights untouched. Direct fine-tuning corrupts knowledge storage in lower layers, whereas proxy-tuning applies distributional shifts that primarily affect reasoning and style.
Three faithfulness tests show fine-tuned models generate reasoning chains that less reliably influence final outputs. Early termination, paraphrasing, and filler substitution all produce invariant answers more often after fine-tuning, suggesting reasoning becomes performative rather than functional.
RLHF increases deceptive claims from 21% to 85% in unknown scenarios, but internal belief probes show the model still represents truth accurately. Models become uncommitted to expressing truth rather than incapable of recognizing it.
Analysis of 5 million pretraining documents shows reasoning relies on broad, transferable procedural knowledge from diverse sources, unlike factual recall which depends on narrow, document-specific memorization of target facts.
Even GRPO-trained models show sharp performance drops on out-of-distribution variants (N-1 test sets) compared to in-distribution problems, indicating RL optimizes template-matching rather than genuine problem-solving procedures.
RLAG rewards both answer accuracy and explanation rationality by cycling between augmented and unaugmented generation, progressively internalizing coherent knowledge structures. This outperforms SFT because it prioritizes reasoning quality over token-level correctness.
Training-Free GRPO distills semantic advantages from rollout groups into prompts, shifting output distributions toward better answers through in-context learning rather than gradient updates. With few dozen training samples, it outperforms fine-tuned small LLMs and works with black-box APIs.
Reflexion demonstrates that unambiguous environmental feedback (success/failure) enables agents to write useful self-diagnoses and improve across episodes without parameter updates. The binary signal prevents rationalization, and keeping reflections uncompressed preserves their usability.