Can suppressing incorrect behavior alone solve the diversity bottleneck in reasoning RL?
This explores whether the way reasoning RL kills diversity — by penalizing wrong answers — can be fixed by just penalizing them more carefully, or whether suppression is the wrong lever entirely.
This reads the question as asking: if RL's diversity problem comes from rewarding correct answers, can we solve it purely by being smarter about pushing down incorrect behavior — or does diversity require its own positive machinery? The corpus answers fairly cleanly: suppression alone is not a fix, because suppression is the mechanism that creates the bottleneck in the first place. Outcome-based RL that rewards only final-answer correctness sharpens the policy globally — it concentrates probability mass on the trajectories it already favors, and crucially this collapse leaks from solved problems onto unsolved ones, so the model gets narrower exactly where it still needs to explore Does outcome-based RL diversity loss spread across unsolved problems?. The same entropy-collapse signature shows up in search agents, where RL converges on a few reward-maximizing strategies while SFT on diverse demonstrations keeps exploration broad Does reinforcement learning squeeze exploration diversity in search agents?.
A deeper reason suppression can't manufacture diversity: RL isn't adding new behaviors to suppress around. Multiple independent lines of evidence suggest base models already contain latent reasoning strategies, and post-training selects among them rather than creating them Do base models already contain hidden reasoning ability? — RL largely teaches a model *when* to deploy reasoning it already has, not *how* to reason Does RL post-training create reasoning or just deploy it?. Seen that way, suppressing wrong trajectories can only prune an existing repertoire; it has no channel for widening it. Worse, RL tends to amplify a single dominant format inherited from pretraining within the first epoch and quietly extinguish the alternatives Does RL training collapse format diversity in pretrained models?. So pure suppression doesn't just fail to add diversity — it actively narrows the strategy space.
What the corpus suggests instead is that diversity needs its own dedicated mechanism, separate from the correctness signal. One note draws the sharpest version of this: historical exploration (training-time diversity, e.g. UCB-style bonuses) and batch exploration (test-time diversity, e.g. repetition penalties) are *structurally different* and can't be folded into one reward Does outcome-based RL diversity loss spread across unsolved problems?. Adjacent work points the same direction from other angles — structuring a single model's reasoning as an internal dialogue between agents breaks the fixed-strategy rut that monologue reasoning falls into Can dialogue format help models reason more diversely?, and process rewards that score metacognitive moves (planning, exploration, reflection) cut repetitive actions while generalizing better than outcome-only training Can RL agents learn to reason better, not just succeed?.
There's a useful twist worth knowing: the failure modes RL introduces by over-suppressing aren't only about diversity — they're also about *calibration*, and the fixes rhyme. Binary correctness rewards push models toward overconfident guessing because nothing penalizes a confident wrong answer, and the remedy is to add a second objective (a Brier-style proper scoring term) rather than tune the first one harder Does binary reward training hurt model calibration?. Using the model's own answer-span confidence as a reward similarly restores calibration while strengthening reasoning Can model confidence work as a reward signal for reasoning?. The recurring lesson across all of these: a single scalar that only says 'this output was wrong, do less of it' optimizes itself into a corner. Diversity, like calibration, has to be a co-equal objective with its own signal — not a side effect you hope to preserve by suppressing failures more gently.
Sources 9 notes
RL that rewards only final answer correctness sharpens the policy globally, concentrating probability mass on correct trajectories for solved problems while simultaneously reducing diversity on unsolved ones. Historical exploration (training diversity via UCB-style bonuses) and batch exploration (test-time diversity via repetition penalties) require structurally different mechanisms.
RL training compresses behavioral diversity in search agents through the same entropy collapse mechanism documented in reasoning—policies converge on narrow reward-maximizing strategies. SFT on diverse demonstrations preserves exploration breadth, suggesting diversity-preservation techniques are essential for RL search scaling.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
Evidence shows base models already contain reasoning capability in latent form; RL training optimizes deployment timing rather than capability creation. Hybrid models recover 91% of performance gains by routing tokens only, and activation vectors for reasoning strategies pre-exist before any RL.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.
DialogueReason, which structures a single model's internal reasoning as dialogue between distinct agents in separate scenes, overcomes monologue reasoning's fixed-strategy and fragmented-attention weaknesses, especially on tasks requiring multiple problem-solving approaches.
RLVMR uses structured meta-reasoning tags (planning, exploration, reflection, monitoring) with programmatic rewards to train agentic RL. This reduces repetitive actions by 31% compared to outcome-only methods while maintaining better generalization than supervised fine-tuning alone.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.