Asking an AI the same question many times and taking the most common answer: does it help more as a reward for training or as a final vote?
How does training-time consensus differ from inference-time majority voting over samples?
This explores the difference between using "the answer most samples agree on" as a training signal that changes the model, and using the same majority vote at answer time to pick a response without changing the model.
This explores the difference between using "the answer most samples agree on" as a training signal that changes the model, and using the same majority vote at answer time to pick a response without changing the model. The mechanism is the same in both cases: sample a question many times and see where the answers cluster. What differs is what happens to the result. At inference time the vote is a filter. It picks one answer, and the model is exactly as it was before. At training time the vote is a teacher. It becomes the reward that reshapes the model.
The training-time version is Test-Time RL. It sets the majority answer as a stand-in for the correct one and rewards the model for producing it, with no labels and no trained reward model. That creates a bootstrapping loop where the compute spent sampling turns into a better model, and the paper's point is that this works because consensus answers tend to be correct Can models improve themselves using only majority voting?. Voting at inference gets you the consensus answer on that one question. Training on the consensus can make the model produce it more reliably, so you may need fewer samples later.
The two also fail differently. Inference-time voting is a surprisingly hard baseline to beat. It matches or outperforms Best-of-N and sequential revision because it doesn't depend on an unreliable verifier or on the model grading itself Why does majority voting outperform more complex inference methods?. Its weaknesses are structural. It throws away the reasoning in the chains that lost the vote, which is why methods that reason over all the chains together can beat it Does voting discard useful reasoning from losing chains?. It also can't help on problems that need step-by-step accumulation, where a sequential chain of thought can be exponentially better than many short parallel votes When does sequential reasoning beat parallel voting?.
The training-time version carries a risk that inference-time voting doesn't. Majority-vote reward only helps when the model is already right more than about half the time on that kind of prompt. Below that, the consensus is usually wrong, and training on it quietly reinforces the errors When does majority-vote reward actually help test-time learning?. A wrong vote at inference costs you one answer. A wrong vote used as a reward can be baked into the weights. The suggested safeguard is to probe each class of prompt first and train only where the model is already in the favorable regime.
The corpus has no head-to-head comparison of the two. There is adjacent evidence for why training-time methods can do something voting can't: reasoning-trained models keep beating non-reasoning models even when the latter get unlimited inference compute, because training installs a way of reasoning that extra tokens can't substitute for Can non-reasoning models catch up with more compute?. Voting can only choose among what the model can already produce. Training on consensus can change what it produces.
Sources 6 notes
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Test-time RL via consensus succeeds when prior accuracy exceeds ~50%, but below that threshold it silently amplifies wrong answers. Safe deployment requires gated probing per prompt class to confirm the favorable regime before training.
Across benchmarks, majority voting empirically outperforms or matches Best-of-N and sequential revision approaches. Its robustness stems from avoiding unreliable verifiers, poor self-assessment, and unnecessary complexity—making it the right baseline for evaluating reasoning model improvements.
Standard self-consistency voting selects the majority answer but discards intermediate reasoning from non-winning chains. Multi-chain reasoning instead meta-reasons over all chains simultaneously to extract distributed information, improving both task accuracy and producing coherent, auditable explanations.
On structured tasks requiring sequential multi-step reasoning like graph connectivity, chain-of-thought achieves exponentially higher accuracy than parallel voting. The difference emerges because solutions genuinely require accumulating intermediate results sequentially, which short parallel chains cannot achieve.
Show all 6 sources
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Deep Think with Confidence
- Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Ones
- TTRL: Test-Time Reinforcement Learning
- On the Reasoning Capacity of AI Models and How to Quantify It
- Can Large Reasoning Models Self-Train?
- Can Large Language Models Capture Human Annotator Disagreements?
- Psychologically Enhanced AI Agents
- Reasoning Strategies in Large Language Models: Can They Follow, Prefer, and Optimize?