When AI learns by chasing rewards instead of imitating examples, it quietly stops exploring — collapsing onto whichever strategies score highest.
Why does exploration diversity behave differently under reinforcement learning versus supervised fine-tuning?
This explores why reinforcement learning tends to narrow the range of strategies a model tries while supervised fine-tuning on diverse examples keeps that range wide — and why that split isn't absolute.
This explores why exploration diversity shrinks under reinforcement learning but survives under supervised fine-tuning — and the corpus turns out to have a fairly mechanical answer rather than a hand-wavy one. The core reason is that RL and SFT optimize different things. SFT imitates whatever distribution of demonstrations it's shown, so if the demonstrations are varied, the breadth is preserved. RL instead maximizes reward, which mathematically means concentrating probability mass on whatever trajectories score highest — the policy sharpens toward a narrow set of winners. This 'entropy collapse' shows up the same way across very different tasks: search agents Does reinforcement learning squeeze exploration diversity in search agents? and reasoning models both converge on a few reward-maximizing strategies, while SFT on diverse traces keeps the exploration wide.
One of the sharper findings is that this narrowing isn't limited to problems the model has already solved. Outcome-based RL — rewarding only the final answer — sharpens the policy *globally*, so diversity loss on solved problems bleeds over into unsolved ones the model hasn't even cracked yet Does outcome-based RL diversity loss spread across unsolved problems?. That's a real cost: you can lose the exploratory breadth you'd need to solve the next problem while getting better at the last one. A related result shows RL doesn't just narrow behavior, it collapses *format* — within the first epoch it amplifies a single dominant pretraining format and suppresses the alternatives, and which one wins depends on model scale rather than performance Does RL training collapse format diversity in pretrained models?.
But the story isn't 'RL always kills diversity.' The direction actually reverses depending on what the reward incentivizes. RLHF *reduces* lexical diversity in code, where there's a single correct convergence point, but *increases* it in creative writing, where distinctiveness is what gets rewarded Does preference tuning always reduce diversity the same way?. So RL doesn't have an intrinsic bias toward sameness — it faithfully collapses toward whatever the reward signal points at, and when the signal itself is low-variance across a batch, the policy collapses hardest into generic input-agnostic templates Why do language models collapse into generic templates?.
The more interesting turn is that diversity collapse under RL is treatable by changing the reward's shape rather than abandoning RL. If you explicitly reward semantic diversity during training, exploration is *catalyzed* and output quality goes up, not down Can diversity optimization improve quality during language model training?. Keeping rewards vector-valued instead of collapsing them into one scalar gives solutions a natural axis to spread along — spanning a Pareto frontier across test cases or personas produces grounded diversity instead of external regularization Can reward vectors be the hidden source of solution diversity?. And training on the messy full exploration path — failures, backtracking, recovery — rather than only clean winning solutions preserves the search behavior that pure outcome rewards would prune away Can models learn better by training on messy exploration paths?.
The thing you didn't know you wanted to know: the SFT-vs-RL difference isn't really about the training algorithm at all — it's about what each one is told to copy. SFT copies a *distribution* (breadth in, breadth out); scalar-reward RL copies a *ranking* (mass flows to the top). The fix for RL's collapse isn't to make it 'more like SFT' but to make the reward itself multi-dimensional, so 'maximize reward' and 'stay diverse' stop being in tension. Worth noting the flip side too: SFT's fidelity to its demonstrations is also a ceiling — agents trained only on static expert data can't learn beyond what the curators imagined Can agents learn beyond what their training data shows?, which is exactly the exploration RL was supposed to buy you.
Sources 9 notes
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
RLHF reduces lexical-syntactic diversity in code generation but increases it in creative writing. The direction depends on what each domain incentivizes: code rewards convergence toward correct solutions, while creative writing rewards stylistic distinctiveness.
When within-prompt reward variance is low, task gradients weaken and regularization dominates, pushing policies toward generic outputs. SNR-Aware Filtering—selecting high-variance prompts before updates—recovers performance across tasks and scales.
Show all 9 sources
DARLING jointly optimizes for quality and semantic diversity using a learned classifier, finding that diversity rewards catalyze exploration and produce higher-quality outputs than quality-only baselines across both creative and mathematical tasks.
Vector Policy Optimization shows that rewards decomposed per test-case, criterion, or persona provide an inherent diversity structure. Training solutions to span the Pareto frontier across these dimensions produces competent diversity grounded in real task trade-offs rather than external regularizers.
Research shows that training on messy trajectories—failed attempts, self-correction, and backtracking—teaches more robust reasoning than training only on shortcut solutions. This approach models o1-style deep reasoning as search internalization rather than solution memorization.
Agents trained on static expert datasets cannot learn from their own failures or generalize beyond demonstrated scenarios because they never interact with environments during training. Competence is capped by what curators imagined, not by agent capacity.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Jointly Reinforcing Diversity and Quality in Language Model Generations
- Vector Policy Optimization: Training for Diversity Improves Test-Time Search
- Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
- Mechanistically Interpreting the Role of Sample Difficulty in RLVR for LLMs
- Outcome-based Exploration for LLM Reasoning
- RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
- Evaluating the Diversity and Quality of LLM Generated Content
- RAGEN-2: Reasoning Collapse in Agentic RL