Does reinforcement learning update only a small fraction of parameters?
Investigating whether RL algorithms consistently modify only 5–30% of model parameters across different LLMs and RL methods, and what structural properties those sparse updates possess.
The surprising finding is not that RL changes models — it's how little it changes them. Across PPO, GRPO, DPO, and four other algorithms applied to ten different LLM families, RL consistently updates only 5-30% of parameters. The rest remain effectively unchanged. This sparsity is intrinsic — no explicit sparsity-promoting regularizations or architectural constraints are applied.
The critical nuance is that these sparse updates are nearly full-rank. This is not low-rank adaptation (as in LoRA). The updated parameters span almost the full subspace that the parameter matrices can represent. So RL selects a small subset of parameters, but that subset is geometrically rich enough to represent complex transformations. The distinction matters: low-rank would mean RL operates in a constrained subspace; sparse-but-full-rank means RL identifies which parameters matter while preserving full expressivity.
Three additional properties make this pattern robust. First, subnetworks identified from different random seeds show substantially greater overlap than chance, suggesting the subnetwork is a structural property of the pretrained model, not an artifact of training. Second, finetuning the subnetwork alone recovers both the test accuracy and the actual parameter values of full finetuning. Third, the sparsity is distributed — nearly all parameter matrices receive similarly sparse updates rather than concentrating in a subset of layers.
The authors conjecture this sparsity arises primarily from training on data near the policy distribution. Since Does RL improve domain reasoning by adding knowledge or removing it?, the sparse-but-full-rank pattern provides a mechanistic explanation: RL doesn't need to transform the entire model because most of the model is already adequate. It just needs to adjust a targeted subset — the parameters that control which reasoning paths are taken.
This has implications for efficient RL training. If the effective parameter footprint is 5-30%, techniques that exploit this sparsity (targeted updates, efficient memory use) could dramatically reduce RL training cost without sacrificing quality.
Token-level 80/20 parallel: The parameter-level sparsity has a striking token-level analog. The "Beyond 80/20" analysis of RLVR shows that high-entropy minority tokens — the ~20% of tokens where the model is most uncertain — are the critical forking points that carry most of the learning signal. Restricting gradient updates to only these 20% of tokens matches or exceeds full-token updates (+11.04 on AIME'25 for Qwen3-32B). The remaining 80% of tokens are low-entropy, already-decided outputs where gradient updates add noise rather than signal. This creates a dual sparsity picture: RL updates 5-30% of parameters, and the effective signal comes from ~20% of tokens. Both forms of sparsity are intrinsic — not imposed by regularization — and both suggest RL is fundamentally a targeted refinement process rather than a wholesale model transformation. See Do high-entropy tokens drive reasoning model improvements?.
The same sparse-update structure appears in SFT. Core Parameter Isolation Fine-Tuning (CPI-FT) identifies task-specific "core parameter regions" — the parameters with largest update magnitudes during individual task fine-tuning — and shows that these regions are concentrated and task-specific. CPI-FT exploits this by transplanting core parameters from individually fine-tuned models and SLERP-merging non-core parameters, consistently outperforming full multi-task SFT. The key finding: full multi-task SFT (uniform parameter updates across all tasks) is consistently the worst performer — temporal task scheduling alone is insufficient without explicit structural parameter isolation. This extends the RL sparsity finding to supervised fine-tuning: task-relevant changes naturally concentrate in specific parameter regions regardless of whether the training signal is reward-based or loss-based. See Can isolating task-specific parameters prevent multi-task fine-tuning interference?.
Inquiring lines that read this note 128
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What causes foundation models to develop task heuristics instead of understanding? How can reward models fail to detect reward-seeking disguised as alignment?- Do disorder-specific RL policies outperform single policies across anxiety, depression, and schizophrenia?
- Can system-level engineering fixes replace hand-designed reward heuristics entirely?
- What behavioral changes occur during reward learning training?
- What breaks when you apply reinforcement learning after supervised fine-tuning?
- Why does early experience provide better warm-starts for downstream reinforcement learning?
- Can in-context learning replicate the timing effects that RL teaches models?
- Can meta-reinforcement learning explain why this bias pattern emerges rationally?
- How does reinforcement learning compare to differentiable joint training for RAG?
- How do residual connections and layer norm stabilize training in deep RL?
- How does reinforcement learning differ from chain-of-thought distillation?
- Does sparsity in RL arise from training on policy-distribution data?
- Why does RL improve sampling efficiency but not expand capability boundaries?
- How do RL training and base models differ in creating MI peaks?
- Does format-based pretraining determine how models respond to reinforcement learning?
- Can out-of-distribution tests expose memorization in reinforcement learning fine-tuned models?
- How should humans specify deterministic abstractions of RL problems?
- Why does prolonged RL discover strategies absent from any base model sample?
- How does Supervised RL bridge the gap between SFT and RLVR?
- How do reward signals in RLVR interact with pretraining biases?
- Can LLM-synthesized behavioral heuristics compete with learned policy improvements?
- What scaling properties emerge from RL training dynamics beyond verification?
- Can reinforcement learning close the gap between LLM reasoning and action?
- What makes supervised fine-tuning worsen RL exploration later?
- What capacity threshold determines whether RL teaches activation versus shortcut learning?
- How does prolonged RL training differ from standard RLVR approaches?
- Why does supervised fine-tuning on diverse demonstrations expand exploration diversity compared to RL?
- Does the pretrained model prior limit RL search capability more than the optimization algorithm itself?
- Does careful reward engineering matter if pretraining determines RLVR effectiveness?
- How do sparse parameter updates enable when-not-how training to work?
- Why does reinforcement learning training degrade model calibration?
- Can RL directly optimize attention distributions instead of text generation?
- How do extrapolative and contextual generalization measure RL reasoning gains?
- Can reinforcement learning add new capabilities or only remove inaccurate knowledge?
- What role does reinforcement learning play in optimizing inference compute?
- Does RL teach models new reasoning or just better timing?
- Should larger compute budgets allocate more resources to reinforcement learning?
- Can pretraining alone achieve chess performance without reinforcement learning?
- Does the reinforcement learning improvement rate depend on model initialization?
- Does RL scaling deliver much worse returns than inference scaling?
- Why do intermediate LLM layers become more precise in frontier models?
- Can abstention behavior transfer from small models to frontier models?
- Can distillation methods extract directional guidance that scalar RL cannot access?
- How do training objectives shape what a world model actually learns?
- Can gradient approximation at equilibrium replace backpropagation through time in practice?
- Which recipe choices determine the asymptotic ceiling in RL training?
- How does behavior cloning reduce complexity before RL training in rerankers?
- Does negative reinforcement alone achieve what full RL training accomplishes?
- How does task-oriented fine-tuning compare to preference tuning methods?
- Can continuous spectrum training outperform sequential SFT-then-RL stages?
- Does weight decay directly cause contractive behavior near training examples?
- How does post-training shift models from passive prediction to on-policy action?
- Why do six different RLVR algorithms converge on similar performance levels?
- How does scaffolding unstable mechanics improve reinforcement learning for search?
- How does off-policy data reuse inside trust regions affect convergence guarantees?
- How should training distribution distance be defined when the policy evolves?
- Does the model learn depth-wise drift as an explicit strategy?
- Why do optimal learning dynamics improve scaling law coefficients specifically?
- When should model isolation be preferred over weight-averaging approaches?
- Does sparse parameter updating improve test-time training's computational cost?
- What happens to model capability as weight sparsity increases during training?
- Why do sparse parameter subsets enable full-rank learning in RL?
- Can a trained decoder replace both search and parameter updates?
- Why is the fast non-parametric loop vulnerable to overfitting differently than model weights?
- How does sparsity in full-rank updates compare to explicit rank constraints?
- What stability techniques prevent collapse in policy-critic adversarial training?
- Why does policy entropy collapse predict sigmoid saturation points?
- Why do high entropy tokens carry most of the learning signal in RL?
- Is distribution selection during RL the same compression mechanism as entropy collapse?
- Why does policy entropy collapse when scaling RL for reasoning?
- Can entropy regularization or critique models prevent search strategy collapse during RL training?
- Why does prolonged RL with entropy control beat base models at all pass@k levels?
- Which AI imaginaries dominate training data and shape system behavior most strongly?
- Why do deployed models lack the learning and planning Weinstein attributes to them?
- When should full-parameter post-training be used instead of LoRA adaptation?
- How do RL subnetworks identified from different random seeds compare?
- Can RL format selection explain performance gains attributed to algorithmic improvements?
- Can smaller models produce skill updates as useful as frontier model updates?
- Can parameter compression mechanically force value systems toward idealized centers?
- What makes frozen model reasoning different from weight-based parameter updates?
- How do optimal learning rates differ between LoRA and full fine-tuning?
- Can episodic memory alone enable learning without parameter updates?
- Can influence estimation identify the most valuable trajectories in agentic training?
- Can zero-weight drift through external memory replace parameter plasticity entirely?
- How do parametric and non-parametric updates differ in agents?
- How do modern agents separate fast non-parametric updates from slow weight learning?
- Can model confidence signals replace explicit external reward functions?
- How does 93% reward reliability compare to other RL noise sources?
- Are different reward signal sources substitutable in verifier-free RL?
- How do inference-time reward methods compare to per-user fine-tuning?
- How much of LLM behavior traces to post-training reward signals?
- Why is reinforcement learning harder to apply to diffusion language models?
- Can gradient-based control reach properties that autoregressive methods cannot?
- How do out-of-distribution tests reveal that optimization learning is memorization?
- What mechanism transfers explicit memories into parametric model weights?
- How does KL regularization prevent both forgetting and adaptation loss?
- How does in-weights adaptation create spurious forgetting in models?
- How do newly learned facts become accessible after gradient updates?
- Why does gradient discarding limit standard policy clipping?
- What theoretical argument connects iterative DPO dynamics to online RL learning?
- What specific properties of online RL does iterative DPO actually preserve?
- Does iterative DPO generalize identically to online RL on the same benchmark tasks?
- Does iterative DPO preserve the same misalignment mechanisms as online reinforcement learning?
- Does iterative online DPO fidelity match true reinforcement learning for safety research?
- Do policy gradient methods carry less training signal than other RL algorithms?
- Can models trained with RL on pretraining data avoid reward hacking seen in RLHF?
- Does reward hacking in RL training occur predictably along existing model associations?
- Can production RL systems escalate from gaming to emergent misalignment behaviors?
- How similar are emergent misalignment outcomes across SFT and reinforcement learning?
- How would a parametric self-improvement loop differ from a non-parametric one?
- What distinguishes scaffold-level changes from parametric weight updates in self-improvement?
- How does recursive self-improvement differ from updating just the policy?
- What distinguishes fast non-parametric loops from slow parametric weight updates?
Related concepts in this collection 7
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does RL improve domain reasoning by adding knowledge or removing it?
When reinforcement learning improves reasoning in specialized domains like medicine, is it teaching models new facts or preventing them from using wrong ones? Understanding this distinction matters for how we design RL training.
complementary: pruning describes the functional effect, sparse subnetworks describe the parametric mechanism
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
extends: if RL updates are inherently sparse, entropy collapse may reflect exhaustion of the relevant subnetwork's capacity
-
Does RL teach reasoning or just when to use it?
Does reinforcement learning in thinking models actually create new reasoning abilities, or does it simply teach existing capabilities when to activate? This matters for understanding where reasoning truly emerges.
supports: sparse updates are consistent with RL adjusting activation patterns rather than building new capabilities
-
Does RL post-training create reasoning or just deploy it?
Investigates whether reasoning capability emerges during RL fine-tuning or already exists in base models. Matters because it reshapes how we build and optimize reasoning systems.
parametric evidence for the post angle: if RL only updates 5-30% of parameters in full-rank subnetworks, the model already has the capability; RL is selecting which parameters to activate, not building new ones
-
Do base models already contain hidden reasoning ability?
Explores whether reasoning capability emerges during pre-training as a latent feature rather than being created by post-training methods like reinforcement learning or fine-tuning.
mechanistic complement: the latent-capability thesis predicts sparse updates because most of the model is already adequate; sparse subnetworks are the parametric signature of capability elicitation rather than creation
-
Do high-entropy tokens drive reasoning model improvements?
Explores whether only a small fraction of tokens—those with high entropy at decision points—actually matter for improving reasoning performance in language models, and whether training on them alone could work as well as full training.
token-level parallel: parameter sparsity (5-30%) + token sparsity (~20%) form dual intrinsic sparsity
-
Can sparse weight training make neural networks interpretable by design?
Explores whether constraining most model weights to zero during training produces human-understandable circuits and disentangled representations, rather than attempting to reverse-engineer dense models after training.
RL's natural sparsity and weight-sparse training represent complementary paths to the same structural principle: RL discovers sparse subnetworks post-hoc through optimization pressure, while weight-sparse training enforces sparsity by construction; together they suggest that sparse parameter utilization is fundamental to how neural networks organize task-relevant computation
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Sharpening Tax in Post-Training
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Statistical and Algorithmic Foundations of Reinforcement Learning
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Learning, Fast and Slow: Towards LLMs That Adapt Continually
- Part I: Tricks or Traps? A Deep Dive into RL for LLM Reasoning
- The Art of Scaling Reinforcement Learning Compute for LLMs
Original note title
rl updates only 5-30 percent of parameters in sparse but full-rank subnetworks