Letting an AI vote against itself skips paid labels and reward models, but what does all that repeated answering cost?
What compute costs does majority-vote consensus sampling add versus supervised training?
This explores what extra compute majority voting adds (sampling the same question many times and keeping the most common answer) and how that compares with paying for supervised training on ground-truth labels.
This explores what extra compute majority voting adds, meaning many samples per question, and how that compares with paying for supervised training on labeled answers. The corpus has no head-to-head dollar or GPU-hour comparison, so I can't give you a number. What it does show is where the cost moves. Voting swaps the cost of labels and reward models for the cost of repeated sampling.
The extra compute is the repeated samples. Every vote is a full generated answer, so cost grows with the number of samples per question. In test-time RL, that repeated sampling is the reward signal itself. Can models improve themselves using only majority voting? needs no ground-truth labels and no trained reward model, so you skip the annotators and the reward-model training run and pay in sampling instead. Self-distillation follows the same trade. Can a model's own consensus replace ground truth labels? matched or beat supervised methods on five benchmarks, and it trains only on rollouts where the model disagrees with itself. The teaching effort goes to the disagreements. Voting also avoids running a separate verifier, which is part of why Why does majority voting outperform more complex inference methods? finds it a robust baseline against fancier inference methods.
The corpus suggests the sampling bill can be cut. Does step-level confidence outperform global averaging for trace filtering? reaches comparable gains to naive majority voting with far fewer generated traces, by spotting weak reasoning steps early and stopping. That implies naive voting wastes a lot of samples. There is also a way to pay for consensus once instead of on every query. Can models trained on many imperfect experts outperform everyone? shows that a model trained on many imperfect experts already absorbs a majority-vote consensus, and low-temperature sampling pulls it out in a single pass. Training buys you the consensus one time. Explicit voting rebuys it at every question.
There are hidden costs, and a ceiling. The voting reward only works when the model is already right more than about half the time. Below that, When does majority-vote reward actually help test-time learning? says it silently amplifies wrong answers, so safe use means probing each prompt class first, which is extra sampling before you train. More votes also can't stand in for training. Can non-reasoning models catch up with more compute? finds that non-reasoning models don't catch up with reasoning models however much inference compute they get. On problems that need step-by-step accumulation, When does sequential reasoning beat parallel voting? shows chain-of-thought beating parallel voting by an exponential margin. Voting is cheap next to labeling, but spending more on it eventually stops helping.
Sources 8 notes
Test-Time RL generates reward signals by majority voting across repeated samples, enabling policy improvement without ground-truth labels or trained reward models. This approach works surprisingly well because consensus answers tend to be correct, creating a bootstrapping loop where test-time compute enables training that improves the model.
Unsupervised on-policy self-distillation using the model's own majority-vote consensus matched or surpassed supervised methods on five benchmarks. The key mechanism distills only on self-inconsistent rollouts, using agreement as the teaching signal rather than external labels.
Across benchmarks, majority voting empirically outperforms or matches Best-of-N and sequential revision approaches. Its robustness stems from avoiding unreliable verifiers, poor self-assessment, and unnecessary complexity—making it the right baseline for evaluating reasoning model improvements.
Local step-level confidence catches reasoning breakdowns that global averaging masks and enables early stopping before traces complete. This approach achieves comparable accuracy gains to naive majority voting with far fewer generated traces, proving trace quality matters more than quantity.
Models trained on diverse experts converge on consensus behavior that outperforms individuals. Low-temperature sampling concentrates outputs on this majority-voted consensus, denoising uncorrelated biases and errors across the training set.
Show all 8 sources
Test-time RL via consensus succeeds when prior accuracy exceeds ~50%, but below that threshold it silently amplifies wrong answers. Safe deployment requires gated probing per prompt class to confirm the favorable regime before training.
Reasoning models persistently outperform non-reasoning models regardless of inference budget because training instills a reasoning protocol that makes additional tokens productive. The gap is fundamentally about deployment mechanisms and training structure, not raw capability.
On structured tasks requiring sequential multi-step reasoning like graph connectivity, chain-of-thought achieves exponentially higher accuracy than parallel voting. The difference emerges because solutions genuinely require accumulating intermediate results sequentially, which short parallel chains cannot achieve.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Deep Think with Confidence
- Can Large Reasoning Models Self-Train?
- TTRL: Test-Time Reinforcement Learning
- Let Me Think! A Long Chain-of-Thought Can Be Worth Exponentially Many Short Ones
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Reinforcement Learning via Self-Distillation
- Transcendence: Generative Models Can Outperform The Experts That Train Them
- Can Large Language Models Capture Human Annotator Disagreements?