When an AI keeps learning and changing, what do you measure its training data's distance from — where it started, or where it is now?
How should training distribution distance be defined when the policy evolves?
This explores how to measure how far training data or targets are from a model when the model keeps changing during training, so the yardstick can be fixed to the starting point or move with the current policy.
This explores how to measure how far training data or targets are from a model when the model keeps changing during training. The corpus has no settled answer, but it does name the problem. One paper on emergent misalignment defines distance as how far each example sits from the centroid of a fixed dataset in representation space. That definition only works if the data doesn't move. The same paper leaves on-policy RL and distillation, where the model generates its own training data, as future work, even though it cites reward hacking in those settings as key evidence (Does the representational distance account work for on-policy training?). So the measure breaks in the exact setting you're asking about.
The rest of the corpus suggests two ways out: anchor to where the model started, or re-anchor to where it is now. The first is KL drift from the base model. Models trained to stay close to their base distribution, up to 70% closer than parameter-only RL, kept the ability to learn later tasks, while the drifting ones stalled when the domain changed (Does staying close to the base model preserve learning ability?). That makes distance from the start an odometer for lost flexibility. It tells you how far the policy has travelled, not how far the next batch of data is from the policy now.
The second approach makes distance relative to the current policy and actively manages it. TOP-D builds a teacher that sits close to the current student, bounded by a trust region, instead of distilling from a distant fixed target. Because the target moves with the student, gradient variance stays controlled and improvement is guaranteed to be monotonic (Can proximity between teacher and student fix distillation instability?). Here distance is a quantity you set each step, not a fixed property of the dataset.
The corpus also shows that the space you measure in matters. In parameter space, RL touches only 5–30% of weights, in nearly full-rank subnetworks that are almost identical across random seeds (Does reinforcement learning update only a small fraction of parameters?). A whole-model weight distance would therefore be dominated by a small, predictable slice. In behavior space, the policy's output distribution narrows. RL amplifies one pretraining format within the first epoch while collapsing the others (Does RL training collapse format diversity in pretrained models?). Outcome-based RL carries diversity loss from solved problems over to unsolved ones (Does outcome-based RL diversity loss spread across unsolved problems?). Search agents show the same squeeze (Does reinforcement learning squeeze exploration diversity in search agents?). My inference is that a drift measure computed only on the prompts being trained on would miss this spillover.
A third definition treats distance as difficulty relative to what the policy can currently do. Within-prompt reward variance is measured from the policy's own rollouts, so it updates as the policy improves. When it's too low, task gradients weaken and the policy slides toward generic templates (Why do language models collapse into generic templates?). One such statistic can both weight tokens and filter out prompts that carry no signal (Can one statistical measure serve dual purposes in RL training?). At the far end, problems that are nearly impossible for the current policy teach shortcuts that contaminate existing skills (Do overly hard RLVR samples actually harm model capabilities?). What needs measuring also shifts over training, from execution tokens first to planning tokens later (Does RL training follow a predictable two-phase learning sequence?), so a single averaged number may hide the important part. No note here unifies these into one definition, but together they point toward a moving reference, current-policy rollouts bounded by a trust region, with a fixed base-model anchor as a drift budget.
Sources 11 notes
The paper's emergent misalignment framework uses distance-to-centroid, which requires a fixed dataset, but leaves on-policy RL and distillation as future work despite citing reward hacking in those settings as key evidence of misalignment.
FST-trained models stay up to 70% closer to their base distribution than parameter-only RL, and this reduced drift preserves the model's ability to learn subsequent tasks effectively. Parameter-only approaches stall when task domains change, while low KL drift enables sustained adaptation.
TOP-D constructs a close teacher instead of distilling from a distant target, bounded by a trust region. This controls gradient variance, guarantees monotonic improvement, and outperforms standard distillation with zero computational overhead.
Across seven RL algorithms and ten LLM families, RL induces intrinsic parameter sparsity of 5–30% without explicit regularization. Critically, these sparse updates are nearly full-rank and nearly identical across random seeds, indicating structural rather than arbitrary parameter selection.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
Show all 11 sources
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
DRO reuses a single self-supervised statistic at two aggregation levels: token-level weighting in dense rewards and query-level filtering to discard degenerate comparisons. This dual use achieves 2–3× faster training with better stability on unverifiable tasks.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Across eight models, RL training consistently shows a first phase where execution correctness drives learning, followed by a second phase where strategic planning becomes the bottleneck. Planning token entropy increases while execution entropy stabilizes, and concentration of optimization on planning tokens yields significant performance gains.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- The Art of Scaling Reinforcement Learning Compute for LLMs
- ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
- RAGEN-2: Reasoning Collapse in Agentic RL
- Reinforcement Learning for Reasoning in Large Language Models with One Training Example
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Reinforcement Learning Finetunes Small Subnetworks in Large Language Models