Why does evaluating multiple candidates work better than judging one answer?
This explores why generating and comparing several candidate answers — then having a judge or reward model weigh them against each other — tends to beat scoring a single answer in isolation, and what mechanisms in the corpus explain that gap.
This explores why evaluating multiple candidates outperforms judging one answer — the difference between scoring in isolation and scoring by comparison. The short version the corpus keeps circling back to: judging well is itself a reasoning task, and comparison gives the judge more signal to reason over. When a model sees several solutions side by side, the contrast between a correct chain and a flawed one becomes the activation signal — that's exactly what Can a single problem unlock reasoning through solution critique? found, where exposure to correct-versus-incorrect reasoning on even a single problem was enough to unlock reasoning ability, no reinforcement learning required. A lone answer offers no such contrast.
A second thread: the act of evaluating gets dramatically better when the judge reasons rather than classifies. Can judges that reason about reasoning outperform classifier rewards? shows judges that produce a reasoning chain about the candidate's reasoning beat classifier-style reward models, with far less training data, and Can reward models benefit from reasoning before scoring? reports three independent teams discovering that letting reward models think before scoring raises their capability ceiling beyond outcome-only evaluation. Comparison is what reasoning judges are good at — laying multiple candidates next to each other gives the thinking judge something to discriminate. It also makes judgment more honest: Can reasoning during evaluation reduce judgment bias in LLM judges? finds that judges trained to reason through evaluations shed the lazy heuristics (verbosity, position, authority bias) that a single-answer scorer leans on when it has nothing to compare against.
There's a subtler payoff at training time, not just test time. Do critique models improve diversity during training itself? argues that comparing and critiquing many candidate steps counteracts 'tail narrowing' — the tendency for self-training to collapse onto a few high-probability answers. Judging multiple candidates keeps the solution space wide, which matters more than any single accuracy bump. But quantity isn't free: Does step-level confidence outperform global averaging for trace filtering? shows that *how* you compare candidates matters as much as how many — step-level confidence catches reasoning breakdowns that crude majority voting over whole answers masks, hitting the same accuracy with far fewer traces.
The corpus also marks the boundary where comparison stops helping. When does sequential reasoning beat parallel voting? is the sharp counterpoint: on genuinely compositional problems — graph connectivity, multi-step accumulation — generating many short parallel candidates and voting loses badly to one long sequential chain, because the answer requires building intermediate results in order that no short candidate can reach. So 'evaluate many' beats 'judge one' when the task admits diverse independent attempts, but not when the task is irreducibly sequential.
Worth a caution flag for anyone designing such systems: more candidates and more comparison can fool the *human* in the loop too. Do users trust citations more when there are simply more of them? found people trust answers with more citations regardless of whether those citations are relevant — a reminder that the volume of supporting material is a trust heuristic that decouples from quality. The lesson across all of this: comparison works because it turns evaluation into a reasoning problem with contrast to reason over — but the gains come from the discrimination, not the count.
Sources 8 notes
Critique Fine-Tuning achieves reasoning activation comparable to RLVR using only one problem and teacher-generated critiques of varied solutions, with no reinforcement learning. This demonstrates that exposure to correct versus incorrect reasoning on a specific problem is the sufficient activation signal.
StepWiser demonstrates that training judges to produce reasoning chains about policy reasoning—rather than classify steps—yields better judgment accuracy and data efficiency. Independent confirmation from GenPRM and ThinkPRM shows generative PRMs outperform discriminative ones with orders of magnitude less training data.
Three independent teams (RRM, RM-R1, DeepSeek-GRM) discovered that adding chain-of-thought reasoning before reward scoring enables adaptive test-time compute scaling for evaluation. Reasoning-based approaches raise the capability ceiling of reward models beyond what outcome-based evaluation achieves.
Training judges with reinforcement learning to reason about evaluations—by converting judgment tasks into verifiable problems with synthetic data pairs—produces judges that think through their decisions rather than relying on exploitable surface features, directly mitigating authority, verbosity, position, and beauty bias.
Step-level critique in the training loop counteracts tail narrowing and maintains solution diversity across self-training iterations. This training-time benefit—preventing premature convergence—is more fundamental than test-time accuracy gains.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
On structured tasks requiring sequential multi-step reasoning like graph connectivity, chain-of-thought achieves exponentially higher accuracy than parallel voting. The difference emerges because solutions genuinely require accumulating intermediate results sequentially, which short parallel chains cannot achieve.
Analysis of 24,000 Search Arena interactions shows irrelevant citations boost user preference (β=0.273) nearly as much as relevant citations (β=0.285), indicating citation count functions as a decoupled trust heuristic.