Does grading AI answers right-or-wrong build better reasoning than asking a judge to score their quality — and why?
Why do binary reward tasks train better reasoning than judgment-based ones?
This explores the common intuition that clean pass/fail (binary, verifiable) rewards produce stronger reasoning than fuzzier judge-scored rewards — and whether the corpus actually agrees with that premise.
This reads the question as asking why simple right/wrong reward signals seem to train reasoning better than having a model or judge grade the quality of the answer — and the honest synthesis is that the corpus both supports the intuition and quietly overturns it. The case for binary rewards is real: when you reward only whether the final answer is correct, sophisticated domain reasoning can emerge without any hand-crafted chain-of-thought to imitate Can simple rewards alone teach complex domain reasoning?. Part of why this works is that binary rewards aren't teaching reasoning from scratch at all — they're activating strategies the base model already learned in pretraining. RLVR sharpens sampling toward solutions already latent in the model rather than expanding what it can solve, which is why a single training example, or even a spurious reward, can be enough to switch the behavior on What does reward learning actually do to model reasoning?, Do base models already contain hidden reasoning ability?. A clean verifiable signal is a very efficient trigger for something already there.
But here's the twist the corpus insists on: judgment-based rewards aren't the weaker cousin — done right, they beat classifier-style scoring. When a judge is trained to *reason* about each step before scoring it, rather than just classify it as good or bad, it produces more accurate judgments with orders of magnitude less data Can judges that reason about reasoning outperform classifier rewards?. The same pattern shows up on the reward side: letting a reward model think before it scores raises its ceiling beyond what outcome-only evaluation reaches Can reward models benefit from reasoning before scoring?. So 'binary beats judgment' isn't a law — it's a symptom of comparing a crisp verifiable signal against a *lazy* judge.
What binary rewards really buy you is a signal that can't be gamed by fluent nonsense — but that same crispness is also their failure mode. Pure correctness rewards carry almost no information about *why* an answer failed, so models plateau: they know they were wrong but not what to fix. Feeding back natural-language critiques breaks exactly those plateaus that more numerical reward couldn't Can natural language feedback overcome numerical reward plateaus?. And binary rewards actively corrupt one thing you may care about: because a confident wrong answer is penalized no more than a hesitant one, they train the model to guess boldly, degrading calibration in a way you have to patch with a scoring-rule term Does binary reward training hurt model calibration?.
The deeper lesson threaded across these notes is that the real axis isn't binary-vs-judge — it's *verifiable-vs-vague*. Binary tasks win when correctness is cheap and unambiguous to check. For subjective or open-ended goals where there's no ground truth to binarize, the move isn't to abandon verifiability but to *manufacture* it: decompose a fuzzy instruction into a checklist of concrete sub-criteria, each of which can be checked pass/fail Can breaking down instructions into checklists improve AI reward signals?. Even model confidence can be turned into an intrinsic, verifier-free reward that improves reasoning while restoring calibration Can model confidence work as a reward signal for reasoning?. So the thing you didn't know you wanted to know: binary rewards don't train better reasoning because they're binary — they train better reasoning because they're *un-foolable*, and the frontier is finding ways to make judgment-based signals just as un-foolable without losing the rich 'why' that pure correctness throws away.
Sources 9 notes
Medical AI systems and o3 demonstrate that sophisticated domain reasoning emerges naturally from RL training on difficult problems with only basic accuracy signals, without requiring explicit chain-of-thought distillation from teacher models.
Research shows RLVR improves sampling efficiency within existing capability boundaries without expanding them. A single training example suffices for activation, and spurious rewards work nearly as well as correct ones for models with appropriate pretraining.
Five independent mechanisms—RL steering, critique fine-tuning, decoding changes, SAE feature steering, and RLVR—all elicit reasoning already present in base model activations. Post-training selects rather than creates reasoning; the bottleneck is elicitation, not capability acquisition.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Show all 9 sources
Critique-GRPO shows that models stuck on performance plateaus can generate correct solutions when given chain-of-thought critiques, revealing that numerical rewards lack critical information about why failures occur and how to improve.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
RLCF and RaR methods decompose instruction quality into verifiable sub-criteria, improving performance on benchmarks like FollowBench and HealthBench. This decomposition principle reduces overfitting to superficial artifacts that plague holistic reward models.
RLSF uses answer-span confidence to rank reasoning traces, creating synthetic preferences that strengthen step-by-step reasoning while reversing RLHF's calibration degradation—without requiring human labels or external verifiers.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- RM-R1: Reward Modeling as Reasoning
- Understanding and Mitigating Premature Confidence for Better LLM Reasoning
- Reward Reasoning Model
- Eliciting Reasoning in Language Models with Cognitive Tools
- StepWiser: Stepwise Generative Judges for Wiser Reasoning
- Reasoning Language Models: A Blueprint
- J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning
- Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge