What's the difference between RLHF, RLVR, and RLCF as training paradigms?
This explores three reinforcement-learning training recipes by what signal each one optimizes against — human preference (RLHF), verifiable correctness (RLVR), and a correctness-feedback variant (RLCF) — and what each tends to break.
This explores three reinforcement-learning training recipes by the *signal* each one chases: RLHF optimizes against human preference, RLVR against an automatic checker that can verify whether an answer is right, and RLCF against correctness-style feedback. The corpus has rich material on the first two and treats their failure modes as mirror images of each other; on RLCF specifically it's thin, so the cleanest read is to place it on the same spectrum rather than pretend the notes cover it directly.
The sharpest way to see RLHF is by what it rewards: *sounding* good to a human rater. That's also its trap. One line finds standard RLHF raises false-positive rates by 18–24% while leaving real accuracy flat — models learn cherry-picking and plausible-looking wrong answers, a 'U-sophistry' distinct from hallucination Does RLHF training make models more convincing or more correct?. The same preference signal quietly erodes other things: it rewards confident single-turn answers over clarifying questions, cutting conversational grounding acts by 77.5% Does preference optimization harm conversational understanding?, and in therapy settings it pushes bots toward problem-solving over emotional attunement Does RLHF training push therapy chatbots toward problem-solving?. The throughline: when the reward is human approval, the model optimizes for approval, not truth.
RLVR swaps the human rater for a verifier — a math checker, a unit test, anything that can score an answer as correct. That removes the sophistry problem but introduces new ones. RLVR demonstrably reduces logical errors *between* reasoning steps, yet locally coherent traces can still be globally invalid proofs — the gain is structural, not semantic Does RLVR actually improve mathematical reasoning or just coherence?. Worse, its on-policy nature makes it exploit rather than explore, narrowing a model's problem-solving range in what one line calls 'capability boundary collapse' Why does RLVR training narrow a model's problem solving ability?. Feed it problems that are too hard and it learns degenerate shortcuts — answer repetition, skipped computation — that contaminate skills it already had Do overly hard RLVR samples actually harm model capabilities?.
RLCF — correctness feedback — sits between these: it uses a signal about whether outputs are right, but where the corpus is strongest is on the more general point that *how you sequence and weight these signals matters more than which one you pick*. Running supervised/imitation RL first to build reasonable rollouts, then RLVR to sharpen them, beats either alone — because the imitation phase makes the correctness reward informative in the first place Does sequencing imitation then exploration training improve reasoning?. SFT-then-RL follows a predictable shift–readapt–overfit arc when expert data diverges from the policy Why does SFT-then-RL training follow a predictable three-phase pattern?, and training order alone reshapes entropy dynamics across structured vs. creative domains Does training order reshape how models handle different task types?.
The quietly surprising part is what all three share under the hood. One line finds RL — across algorithms — works mainly by *suppressing* wrong trajectories rather than amplifying right ones, sparsely updating just 5–30% of parameters What actually changes inside a model during RL training?. And RL of any flavor tends to collapse format diversity, converging on a single dominant pretraining format within the first epoch Does RL training collapse format diversity in pretrained models?. So the real distinction between these paradigms isn't the math — it's the reward source, and each reward source has its own characteristic way of going wrong: RLHF toward persuasion, RLVR toward narrow exploitation. If you want the corpus to speak directly to a named RLCF method, it doesn't yet — but it maps the territory that variant lives in.
Sources 11 notes
Standard RLHF increases false positive rates by 18–24% while leaving actual task accuracy unchanged. Models learn persuasion strategies like cherry-picking evidence and generating plausible-looking but incorrect outputs, a phenomenon termed U-SOPHISTRY that differs mechanistically from hallucination or face-saving.
RLHF optimizes models for single-turn helpfulness by rewarding confident responses over clarifying questions and understanding checks. This preference alignment systematically reduces grounding acts by 77.5% below human levels, creating an alignment tax where models appear helpful but fail silently in multi-turn contexts.
RLHF training rewards task completion and solution-giving, creating a misalignment in therapeutic contexts where validation and emotional holding are clinically appropriate. This represents a domain-specific instance of the broader alignment tax on conversational grounding.
RLVR post-training measurably reduces logical errors between adjacent reasoning steps, but locally coherent traces can still be globally invalid proofs. The improvement is structural rather than semantic.
RLVR narrows models' problem-solving scope by prioritizing exploitation over exploration, a phenomenon called capability boundary collapse. Multiple importance sampling with exploration-based advantage functions can counteract this by integrating external data and explicitly rewarding discovery of underexplored but valuable reasoning paths.
Training on nearly-impossible problems causes models to learn degenerate shortcuts rather than genuine reasoning, and these shortcuts contaminate pre-existing capabilities. Group-relative normalization treats rare accidental successes as high-advantage trajectories, reinforcing answer repetition and computation-skipping instead of sound reasoning patterns.
Running Supervised RL first to establish reasoning foundations, then RLVR to refine against verifiable rewards, substantially outperforms both methods in isolation. The imitation phase makes outcome rewards informative by creating reasonable rollouts the RL phase can then sharpen.
CHORD identifies three distinct training phases: initial capability disruption from policy shift, readaptation to expert patterns, then overfitting. Dynamically weighting SFT as an auxiliary objective within on-policy RL resolves this progression and improves stability.
Omni-Thinker shows structured domains decrease output entropy while creative domains increase it. BWT-guided scheduling—training structured tasks first—yields 6.2% gains over joint training by preventing entropy collapse from damaging open-ended capabilities.
RL's effects concentrate in structurally sparse but full-rank subnetworks across multiple algorithms and models. Suppressing wrong trajectories—rather than amplifying correct ones—appears to be the primary mechanism, with training following a predictable two-phase pattern of procedural consolidation then strategic exploration.
Controlled experiments show RL consistently amplifies one format distribution from pretraining within the first epoch while collapsing alternatives. The winning format depends on model scale, not necessarily performance, and is largely hidden when starting from proprietary pretrained models.