If an AI only trusts answers it repeats consistently, can it still end up confidently wrong?
What role does the self-consistency threshold play in preventing error reinforcement?
This explores whether requiring a model's own answers to agree with each other (a consistency cutoff) can stop it from training on its own mistakes; the corpus has no note about tuning such a threshold directly, so this draws on nearby work about why agreement alone falls short.
This explores whether requiring a model's own answers to agree before trusting them (a consistency cutoff) can stop it from reinforcing its own mistakes. The corpus has no note about picking or tuning that threshold. What it does have is a clear picture of why a threshold on agreement can't do the job alone, and what tends to do the job instead.
The problem is that agreement measures reproducibility, not correctness. Self-consistency works as a reward for training without labels, but Does self-consistency reliably reward correct answers during training? finds that models eventually learn to give confidently wrong answers they can reproduce every time. The link between the proxy reward and being right weakens as training goes on, and the failure looks like improvement because agreement keeps rising. A consistent wrong answer passes any cutoff you set. Training on what passes then makes the model more consistent still, so the mistake gets locked in rather than screened out.
The notes that do prevent reinforcement add a check from outside the model's own agreement. In Can peer models replace external judges for reward signals?, rewards come from a diverse cohort of peer models instead of the model grading itself. This avoids the bias and collapse of self-generated feedback, and it matches the consistency intuition: different models don't share the same blind spots, so their agreement means more. The closest thing to a threshold in the corpus is the held-out validation gate in Does constraining edits make skill learning more stable?. An edit is accepted only if it helps on data the editor never saw. The setup also caps how much can change per step and keeps rejected edits as negative examples. That gate is an independent test of truth, not a vote among the model's own samples.
A second set of notes looks at what the training signal does with the answers, and both point to where confident errors take hold. Reinforcing only what looks right piles probability onto a few answers. In Does negative reinforcement alone outperform full reinforcement learning?, training only on wrong samples matches or beats full RL because it suppresses bad paths while keeping diversity. In Does binary reward training hurt model calibration?, a right-or-wrong reward encourages confident guessing, and adding a Brier score term penalizes being confidently wrong. And Why does self-correction training on offline data fail? shows that self-correction only sticks when the model practices on its own live errors, not on someone else's recorded ones.
Taken together, a self-consistency threshold acts as a filter on confidence, not on truth. It screens out the noisy errors, where the model disagrees with itself. It can't screen out the dangerous ones, where the model is wrong the same way every time. Those need something the model can't agree its way past: a different set of models, held-out data, or a reward that punishes confident wrongness.
Sources 6 notes
Self-consistency works as an intrinsic reward for bootstrapping RL without labels, but models eventually learn to generate confidently wrong but reproducible answers. The proxy reward correlation with correctness degrades over training, creating a failure mode that looks like improvement.
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
SkillOpt's ablations show that adding a textual learning-rate budget, held-out validation gate, and rejected-edit buffer (retaining failed edits as negative feedback) produces more stable and generalizable skill improvement than allowing agents to freely rewrite their own instructions.
Training with only negative samples consistently improves Pass@k across the spectrum, often matching full PPO and GRPO. Negative reinforcement suppresses incorrect trajectories while preserving diversity, whereas positive-only reinforcement degrades higher-k performance by concentrating probability mass.
Binary correctness rewards incentivize high-confidence guessing because they don't penalize confident wrong answers. Adding the Brier score as a second reward term mathematically guarantees joint optimization of accuracy and calibration without trade-off.
Show all 6 sources
SFT on offline correction traces fails because training errors don't match test errors and models collapse into single correction modes. Multi-turn online RL under the model's own error distribution successfully trains self-correction by letting models practice correcting their actual mistakes.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Can Large Reasoning Models Self-Train?
- Training Language Models to Self-Correct via Reinforcement Learning
- Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Future
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- The Surprising Effectiveness of Negative Reinforcement in LLM Reasoning
- Reinforcement Learning with Rubric Anchors
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills
- Reward Reasoning Model