How do residual connections and layer norm stabilize training in deep RL?
This explores how deep reinforcement-learning training stays stable as networks get bigger — but the corpus doesn't tackle residual connections or layer norm as architectural tricks; instead it locates stability in the *learning dynamics* of RL itself.
This reads the question as "what keeps deep RL training from falling apart?" — and here's the honest first thing worth knowing: nothing in this collection is about residual connections or layer normalization as the stabilizing machinery. Those are the textbook answers (skip-connections keep gradients flowing through depth; normalization keeps activations in a sane range). What the corpus has instead is a more interesting story — that in deep RL the real stability problems show up in the *training dynamics*, not the architecture, and they get solved by very different levers.
Start with depth itself, since that's the literal subject. Scaling self-supervised RL to 1000-layer networks doesn't just make things smoother — it unlocks qualitatively new behaviors at specific depth thresholds (walking at depth 16, wall-climbing at depth 256), with gains coming from *both* better exploration and more expressive representations Does network depth unlock qualitatively new behaviors in RL?. So depth in RL isn't merely a thing you have to stabilize against; it's a source of capability — which reframes the whole question.
Where the corpus locates fragility is elsewhere. RL training doesn't smoothly nudge the whole network — it sparsely updates only 5–30% of parameters, in subnetworks that are nearly identical across random seeds, mostly by *suppressing* wrong trajectories rather than amplifying right ones What actually changes inside a model during RL training? Does reinforcement learning update only a small fraction of parameters?. And it moves in two phases: first nailing execution correctness, then shifting the bottleneck to strategic planning Does RL training follow a predictable two-phase learning sequence?. Instability, in this picture, is when those dynamics go wrong — entropy collapsing and killing open-ended ability Does training order reshape how models handle different task types?, or hard samples teaching degenerate shortcuts that contaminate skills the model already had Do overly hard RLVR samples actually harm model capabilities?.
The stabilizers the corpus actually documents are training-side, not architectural. Staying close to the base model's distribution (low KL drift) preserves the model's *plasticity* — its ability to keep learning new tasks instead of stalling when the domain shifts Does staying close to the base model preserve learning ability?. Reusing cross-rollout variance as both a reward weight and a query filter buys 2–3× faster, more stable training by throwing out degenerate comparisons Can one statistical measure serve dual purposes in RL training?. And adding a Brier-score term to binary rewards stops the model from collapsing into confident guessing Does binary reward training hurt model calibration?.
So the thing you didn't know you wanted to know: in this collection, "stabilizing deep RL" is barely about the network plumbing at all. The leverage is in *what you reward, how close you stay to the base model, and which samples you train on* — and depth, far from being the enemy stability fights, is where new behavior comes from.
Sources 9 notes
Scaling to 1000-layer networks in self-supervised RL produces dramatic capability jumps at specific thresholds—depth 16 enables walking, depth 256 enables wall-climbing—driven by synergistic gains in both exploration and expressivity rather than gradual improvement.
RL's effects concentrate in structurally sparse but full-rank subnetworks across multiple algorithms and models. Suppressing wrong trajectories—rather than amplifying correct ones—appears to be the primary mechanism, with training following a predictable two-phase pattern of procedural consolidation then strategic exploration.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.