Letting an AI vote among its own answers at question time is one thing; training it on that vote is another.

This explores what changes when agreement among a model's own sampled answers is used as a training reward, compared with using that agreement at answer time to pick a winner from the samples. The corpus has no note that runs this exact head-to-head. Several notes do pin down the mechanics that separate the two: selection filters a fixed pool of answers, while training rewrites the pool.

Selection is a filter. You sample several answers, keep the one most of them agree on, and the model itself stays the same. The corpus explains why this works: Can models trained on many imperfect experts outperform everyone? shows that models trained on many imperfect experts land on a consensus that beats any individual, because errors that are uncorrelated across sources cancel out. The catch is that the errors have to be uncorrelated. Do different AI models actually produce diverse outputs? found 70+ models giving strikingly similar answers to open-ended questions. Agreement between samples can therefore reflect shared training data and alignment rather than shared truth. Selection still fails gracefully, since a bad vote costs you one answer and the next query starts fresh. Leaving the weights alone also has a benefit of its own: Can decoding-time tuning preserve knowledge better than weight fine-tuning? finds that steering at decoding time preserves pretrained knowledge better than direct fine-tuning, which corrupts knowledge stored in the lower layers.

Training on agreement creates a feedback loop. The reward is computed from the model's own outputs, so whatever the model already does most often becomes what it is paid to do more of. Does RL training collapse format diversity in pretrained models? shows RL amplifying one pretraining format within the first epoch while collapsing the alternatives, and the winner depends on model scale rather than on performance. Do overly hard RLVR samples actually harm model capabilities? shows the same group-relative logic going wrong on problems that are too hard. Rewards attach to whatever happened to succeed, and the model learns shortcuts that leak into skills it already had. That note is about rare lucky wins, not majority votes. The mechanism carries over, though: a confident but wrong majority would be rewarded just as an accidental success is, and the model would learn to be wrong more consistently. Why do ranking systems need to model selection bias explicitly? describes the same trap in ranking systems, where models trained on their own past decisions settle into degenerate equilibria.

The agreement statistic is often more useful for deciding where to train than for deciding what is right. Can one statistical measure serve dual purposes in RL training? reuses one cross-rollout statistic at two levels. It weights individual tokens in a dense reward, and it filters out queries whose comparisons carry no useful signal. The result is 2–3× faster training with better stability on tasks that can't be checked automatically. So the choice is not just agreement-as-training versus agreement-as-selection. Agreement can also work as a routing signal that tells training which examples are worth learning from.

The same split shows up with human agreement. Can crowdsourced votes reliably rank language models? shows that pooling 240K+ pairwise votes to rank models works, because the questions are diverse and the crowd tracks expert raters. Do all annotation responses measure the same underlying thing? warns that feeding those same responses into reward-model training treats genuine preferences, non-attitudes and constructed preferences as one thing, and that contaminates the model. Selection asks which of these is best right now. Training asks what the model should become, and only the second question can shrink the set of answers the model will draw from later.