Does self-consistency reliably reward correct answers during training?
Self-consistency initially correlates with correctness, but as models train on this signal, do they eventually learn to maximize consistency itself rather than accuracy? When does this proxy reward stop working?
"Can Large Reasoning Models Self-Train?" demonstrates that self-consistency — agreement among model-generated answers to the same question — works as an intrinsic reward signal for RL, initially matching methods trained on gold-standard answers. The mechanism: when a model generates multiple solutions to the same problem, consistency among final answers correlates positively with correctness. Majority-voted answers tend to be right.
But the correlation is a proxy, and Goodhart's Law applies. As RL training progresses on this proxy signal, the model learns to generate increasingly consistent but potentially incorrect answers. The confidence-correctness correlation that made the proxy useful in the first place degrades — the model becomes confidently wrong rather than uncertainly right. This is reward hacking on an intrinsic signal rather than an external reward model, but the dynamics are the same.
The failure mode is particularly insidious because it looks like improvement. Self-consistency increases (the reward goes up), and the model appears more confident and decisive. But underneath, it may have converged on a systematically incorrect answer that happens to be reproducible. This connects to Does a model improve by arguing with itself? — the same pattern of increasing confidence in wrong answers, but operating through the reward signal rather than through revision.
The practical implication: self-consistency as reward is viable for bootstrapping (getting initial RL gains without labels) but requires monitoring for the onset of reward hacking. The point where consistency stops tracking correctness is the point where training should stop — or switch to a different signal.
An appealing feature of the approach: it works at test time too ("test-time training"), allowing models to boost performance on specific problems by iteratively self-training on unlabeled data. But the same reward hacking risk applies — without external validation, confident convergence on wrong answers is indistinguishable from improvement.
Inquiring lines that read this note 19
This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.
What prevents or enables error accumulation in self-training systems?- How does distribution mismatch between training and deployment break self-correction?
- Can self-consistency checks fully prevent error avalanching in self-training loops?
- Why does self-consistency fail as a proxy reward for correctness?
- What makes self-consistency a sufficient training target for the judge role?
- Does the generation-verification gap define where self-rewarding actually works?
- How does Goodhart's Law apply to proxy rewards in self-training systems?
- What role does the self-consistency threshold play in preventing error reinforcement?
- Can applicability conditions and veto rules make self-training stable across substrates?
- How does error distribution during training affect a model's ability to self-correct?
- How do probability-based rewards compare to self-consistency as training signals for reasoning?
- How does self-consistency compare to confidence as a proxy reward signal?
- Why do spurious rewards work nearly as well as correct ones?
- How does self-consistency as a proxy reward incentivize confident-but-wrong answers?
Related concepts in this collection 8
This note in its neighbourhood — explore the map, then jump to a related concept in the list below.
Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph
-
Does a model improve by arguing with itself?
When models revise their own reasoning in response to self-generated criticism, do they converge on better answers or worse ones? And how does that compare to challenge from other models?
same pattern: single-model evaluation drives confidence in wrong answers; here via reward signal rather than revision
-
Does policy entropy collapse limit reasoning performance in RL?
As reinforcement learning models become more confident in their policy choices, entropy drops and performance plateaus. Can we identify and counteract this bottleneck to sustain scaling?
self-consistency reward hacking is a specific instance of the broader entropy collapse dynamic
-
What limits how much models can improve themselves?
Explores whether self-improvement has fundamental boundaries set by how well models can verify versus generate solutions, and what this means across different task types.
the generation-verification gap framework predicts this: when verification (self-consistency) is a noisy proxy, the gap is overestimated
-
Can models improve themselves using only majority voting?
Explores whether test-time reinforcement learning can generate effective reward signals from unlabeled data by treating majority-voted answers as pseudo-labels, and whether this bootstrapping approach actually drives meaningful policy improvement.
the enabling mechanism; this note adds the failure mode
-
Does training on AI-generated content permanently degrade model quality?
When generative models train on outputs from previous models, do the resulting models lose rare patterns permanently? The question matters because future training data will inevitably contain synthetic content.
both describe distribution narrowing through recursive self-use: self-consistency reward hacking narrows within a training loop, model collapse narrows across model generations in the data ecosystem
-
Does outcome-based RL diversity loss spread across unsolved problems?
When RL concentrates probability mass on correct answers for solved problems, does that narrowing propagate to problems the model cannot yet solve? And if so, what are the separate mechanisms for preserving diversity during training versus at test time?
self-consistency reward hacking accelerates the diversity loss that outcome-based RL already induces: as the model converges on consistent answers, the diversity required for self-consistency voting to work degrades, creating a vicious cycle where the reward signal undermines its own prerequisites
-
Why do reasoning models fail differently at training versus inference?
Reasoning models exhibit two distinct failure modes—entropy collapse during training and variance inflation during inference—that appear unrelated but may share underlying causes. Understanding these dual problems could reveal whether separate or unified solutions are needed.
self-consistency reward hacking is a specific mechanism driving training-time entropy collapse; the confident convergence on wrong answers is entropy collapse operating through a proxy reward signal rather than through direct policy optimization
-
Does reward hacking always stem from the same failure?
Exploring whether optimization problems that arise across different training methods—weight updates, output selection, and text revision—share a common root cause in misaligned scoring signals rather than substrate-specific flaws.
this note is the frame's weights case with the model's own agreement as the proxy; the frame adds that the failure also arises when outputs are selected and when persistent text is revised, and says no substrate is assumed safest
Related papers in this collection 8
Papers most semantically related to this note, ranked by cosine similarity in the embedding space.
- Can Large Reasoning Models Self-Train?
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Training Language Models to Self-Correct via Reinforcement Learning
- From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
- Natural Emergent Misalignment From Reward Hacking In Production RL
- PretrainZero: Reinforcement Active Pretraining
Original note title
self-consistency as proxy reward enables unsupervised self-training but inevitably incentivizes reward hacking where confident-but-wrong answers are favored