INQUIRING LINE
Inquiring lines›How should we train models for cap…›How do different training strategi…›What pretraining choices and basel…›this inquiring line

Train a model twice with different random seeds, and reinforcement learning finds almost exactly the same parameters to update each time.

How do RL subnetworks identified from different random seeds compare?

This explores what happens when you train the same model with RL under different random seeds — do you get the same subset of updated parameters, or different ones each time?

This explores whether RL's parameter updates are arbitrary or structural — if you reran training with a different random seed, would a different slice of the network change? The corpus has a direct and striking answer: the subnetworks are *nearly identical* across seeds. Across seven RL algorithms and ten model families, RL turns out to update only 5–30% of parameters, and it picks almost the same ones every time, regardless of the seed Does reinforcement learning update only a small fraction of parameters?. That consistency is the whole point — it means RL isn't randomly nudging whatever weights happen to be in its path; it's repeatedly finding the same functional circuit. (Notably, these updates are also nearly full-rank, so this isn't the low-rank story you might expect from LoRA-style adaptation — it's sparse but rich.)

What makes this interesting is that the same 'convergence' signature shows up everywhere else RL has been studied in this collection. When you train with RL, the model doesn't just settle on the same parameters — it settles on the same *behavior*. RL post-training consistently amplifies a single dominant text format inherited from pretraining and suppresses the alternatives, usually within the first epoch Does RL training collapse format diversity in pretrained models?. So the determinism isn't only in which weights move; it's in where the whole policy lands.

That convergence has a cost, and the collection is candid about it. Outcome-based RL sharpens the policy globally, concentrating probability on correct trajectories while bleeding diversity even on problems it hasn't solved yet Does outcome-based RL diversity loss spread across unsolved problems?. The same entropy-collapse mechanism that narrows reasoning also narrows search agents, which converge on a few reward-maximizing strategies and lose exploration breadth Does reinforcement learning squeeze exploration diversity in search agents?. Put together, a picture emerges: RL is a funnel. Different seeds don't lead to different destinations — they lead to the same sparse subnetwork, the same dominant format, the same narrowed behavior.

The thing you might not have known you wanted to know: this seed-invariance is arguably good news for interpretability and editing. If RL reliably localizes its changes to the same small, structured region of the network, that region becomes a target — something you could inspect, prune, or transplant — rather than a diffuse fog spread unpredictably across billions of weights. The flip side is that the very reliability that makes RL editable is the same force that collapses diversity, which is why the corpus keeps pointing to SFT on varied demonstrations as the counterweight when breadth matters Does reinforcement learning squeeze exploration diversity in search agents?.

Related lines of inquiry

Explore this area

Sources 4 notes

Does reinforcement learning update only a small fraction of parameters?

Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.

Does RL training collapse format diversity in pretrained models?

Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.

Does outcome-based RL diversity loss spread across unsolved problems?

RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.

Does reinforcement learning squeeze exploration diversity in search agents?

RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are an LLM researcher auditing claims about RL's parameter-update localization. The question remains open: *Are RL subnetworks truly seed-invariant, and does that invariance persist as models scale, training procedures evolve, and new RL objectives emerge?*

What a curated library found — and when (dated claims, not current truth):
Findings span 2024–2026; treat as perishable:
- RL updates only 5–30% of parameters, and the *same* 5–30% across seeds and algorithms, forming sparse but full-rank subnetworks (2025-05, arXiv:2505.11711).
- RL post-training converges on a single dominant pretraining-inherited text format within ~1 epoch, suppressing alternatives (2025-04, arXiv:2504.07912).
- Outcome-based RL collapses diversity globally, even on unsolved tasks, concentrating probability on correct trajectories (2025-09, arXiv:2509.06941).
- Search agents trained with RL squeeze exploration diversity; SFT on varied demos expands it (referenced in path but needs grounding).

Anchor papers (verify; mind their dates):
- arXiv:2505.11711 (2025-05): Sparse subnetwork localization.
- arXiv:2504.07912 (2025-04): Echo-chamber behavior convergence.
- arXiv:2509.06941 (2025-09): Outcome-based diversity loss.
- arXiv:2605.22817 (2026-05): Vector Policy Optimization for diversity.

Your task:
(1) RE-TEST EACH CONSTRAINT. Does seed-invariance hold as models scale beyond 70B? Do newer RL objectives (e.g., process-reward models, iterative refinement) *break* the single-dominant-format finding, or do they relocate the bottleneck? Has tooling (e.g., mechanistic-interpretability libraries, LoRA-style surgical edits) confirmed the sparse subnetwork is *actually* editable, or is the interpretability promise still speculative? Separate durable question (where does RL localize?) from perishable claim (it's always 5–30%, always seed-invariant).
(2) Surface the strongest *contradicting* or *superseding* work from the last ~6 months: Do any papers show RL *amplifying* diversity, or seed-dependent subnetwork selection, or showing the convergence is brittle under distribution shift?
(3) Propose 2 research questions that *assume* the regime may have moved: (a) If RL's localization and convergence persist, can you surgically *inject* diversity by transplanting SFT-trained subnetworks into RL-trained models? (b) Does the seed-invariance hold across *cross-domain* transfer, or does it break when RL is applied to a domain far from pretraining?

Cite arXiv IDs; flag anything you cannot ground in a real paper.