Training on recycled old data risks instability — does capping how far the model moves each step restore the safety guarantee?
How does off-policy data reuse inside trust regions affect convergence guarantees?
This explores a training-stability question: when you reuse data generated by an older version of a policy (off-policy reuse) rather than freshly sampled data, does bounding how far the policy can move — a trust region — preserve guarantees that training keeps improving rather than diverging?
This explores whether reusing stale, off-policy data is safe when you fence the update inside a trust region — and the corpus's clearest answer is that the trust region is exactly the mechanism that buys back the guarantee you'd otherwise lose. The sharpest illustration is a distillation setup where instead of pulling a student toward a distant, fixed teacher, you construct a *proximal* teacher that stays close to the current student and bound the gap with a trust region Can proximity between teacher and student fix distillation instability?. The payoff isn't just empirical: bounding the divergence controls gradient variance, which is what converts an otherwise unstable process into one with *monotonic* improvement. That's the general principle — off-policy or off-target data is only as dangerous as the distributional distance it introduces, and a trust region is a hard cap on that distance.
The reason variance is the hinge is worth making explicit, because two other notes attack the same problem from the data side rather than the constraint side. One reuses a single self-supervised statistic — cross-rollout variance — both to weight tokens and to *filter out* degenerate queries entirely, discarding comparisons that would inject noise into the update, and reports 2–3× faster, more stable training on exactly the unverifiable tasks where off-policy noise bites hardest Can one statistical measure serve dual purposes in RL training?. Another restructures how rollouts are sampled so that a fixed token budget yields more *distinct* trajectories, sharpening the advantage estimates that off-policy updates depend on Can shared-prefix trees reduce redundancy in agent rollouts?. Trust regions, query filtering, and better sampling are three levers on the same quantity: the reliability of the gradient signal when the data you're learning from isn't fresh.
The uncomfortable counterpoint is what convergence *to* actually means here. A trust region can guarantee you converge stably — but converge toward what? One line of work shows that RLVR-style training doesn't expand the base model's reasoning boundary; it sharpens the sampling distribution toward solutions the model could already reach, improving pass@1 while base models still win at high k Does RLVR actually expand what models can reason about?. Read alongside the distillation result, this suggests trust-region reuse gives you *stable concentration* rather than *new capability* — genuine transfer of new reasoning patterns came from distillation, not from tightening the loop on the model's own outputs. So the convergence guarantee is real but modest: it protects you from divergence, not from converging to a narrow region of what the base model already knew.
There's also a structural reason these guarantees hold more cleanly than you'd expect. RL updates turn out to touch only 5–30% of parameters, in sparse-but-nearly-full-rank subnetworks that are strikingly consistent across random seeds Does reinforcement learning update only a small fraction of parameters?. If the effective update lives in a small, stable subspace, a trust region has a much easier job — it's constraining motion in a low-dimensional region rather than fighting the whole parameter space, which is part of why bounded-divergence methods can promise monotonicity with near-zero overhead.
One last cross-domain framing worth chasing: the same instability that trust regions tame in policy updates shows up as *calibration* damage in reward design. Binary correctness rewards provably push models toward confident guessing, and the fix is an added proper scoring term rather than a constraint on movement Does binary reward training hurt model calibration?. Put next to the trust-region work, it's a reminder that 'convergence guarantee' has two failure modes — the update can be unstable (fix with a trust region) or it can converge stably to a badly-shaped objective (fix the reward). Off-policy reuse inside a trust region solves the first; it does nothing for the second.
Sources 6 notes
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Tree-structured rollouts that branch from shared prefixes produce more distinct trajectories within a fixed token budget than independent chain sampling. This improves advantage estimation statistics and enables longer-horizon tasks within the same compute constraint.
Pass@k analysis shows base models outperform RLVR models at high k, indicating RLVR doesn't expand solvable problems but rather narrows sampling toward solutions already in the base model's distribution. Distillation, by contrast, genuinely transfers new reasoning patterns.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Show all 6 sources
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
- The Art of Scaling Reinforcement Learning Compute for LLMs
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- The Invisible Leash: Why RLVR May Not Escape Its Origin
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models
- Trust Region Policy Distillation
- RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains