Why do sparse parameter subsets enable full-rank learning in RL?
This explores a surprising finding about RL fine-tuning: it rewrites only a small slice of a model's weights (5–30%), yet those edits aren't confined to a few directions — they're full-rank — and the corpus suggests this is because RL reorganizes capabilities the model already has rather than installing new ones.
This explores why RL's weight changes can be both sparse (touching few parameters) and full-rank (spanning many directions) at once — a combination that sounds contradictory until you look at what RL is actually doing inside the model. The anchor result is that across seven RL algorithms and ten model families, RL spontaneously updates only 5–30% of parameters without anyone asking it to, and those updates are nearly full-rank and nearly identical across random seeds Does reinforcement learning update only a small fraction of parameters?. The seed-stability is the tell: if RL were just nudging weights arbitrarily, different runs would pick different parameters. Instead they converge on the same subnetwork, which means the sparsity is structural — RL is finding a specific, pre-existing pathway, not carving a new one.
The 'why' comes into focus when you ask what RL is mechanistically doing. It turns out the dominant move is *suppression*, not amplification: RL mostly works by pushing down wrong trajectories rather than building up correct ones, following a two-phase arc of consolidating procedures first and exploring strategy second What actually changes inside a model during RL training?. Suppressing a behavior the model already knows requires adjusting many directions of the weight space (hence full-rank) but only at the specific parameters that gate that behavior (hence sparse). You don't need to retrain the whole network to change which of its existing tendencies wins.
That framing connects to a second corpus finding that's easy to miss: RL doesn't create format or reasoning behaviors so much as it *selects* among ones pretraining already deposited. RL reliably amplifies a single dominant pretraining distribution within the first epoch while collapsing the alternatives Does RL training collapse format diversity in pretrained models?. If the raw material is already in the base model, the learning problem reduces to routing — turning up one latent capability and turning down its competitors — which is exactly the kind of change a sparse-but-full-rank edit can express.
There's a deeper hint from how networks use sparsity natively. Models already represent unfamiliar or hard inputs sparsely and familiar ones densely, a structure learned during pretraining rather than imposed later Is representational sparsity learned or intrinsic to neural networks?, and they sharpen activations into an even sparser, localized pattern under difficult or out-of-distribution tasks as a stabilizing filter Do language models sparsify their activations under difficult tasks?. So sparse, targeted modification isn't foreign to these models — it's their default organizing principle. RL appears to ride that existing structure rather than fight it.
The payoff you might not have expected: this reframes RL fine-tuning as closer to interpretable surgery than blunt retraining. Work on deliberately training sparse weights shows that sparsity forces modular, human-readable circuits where ablation can prove a circuit is both necessary and sufficient for a task Can sparse weight training make neural networks interpretable by design?. If RL naturally concentrates its edits in a stable, sparse subnetwork, the same logic suggests those subnetworks may be the right unit for understanding — and eventually steering — what RL actually changed in a model.
Sources 6 notes
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
RL's effects concentrate in structurally sparse but full-rank subnetworks across multiple algorithms and models. Suppressing wrong trajectories—rather than amplifying correct ones—appears to be the primary mechanism, with training following a predictable two-phase pattern of procedural consolidation then strategic exploration.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
During pretraining, neural networks develop dense activations for familiar training data and default to sparse representations for unfamiliar inputs. This trend emerges without task-specific fine-tuning and reflects how models consolidate knowledge through exposure.
As task difficulty increases, LLM hidden states become substantially sparser in a localized, systematic way that correlates with task unfamiliarity and reasoning load. This sparsification acts as a selective filter stabilizing performance under OOD shift rather than a failure mode.
Training transformers with sparse weights creates compact, human-interpretable circuits where neurons correspond to simple concepts with clear connections. Ablation studies confirm these circuits are necessary and sufficient for task performance, though scaling beyond tens of millions of parameters while maintaining interpretability remains unsolved.