Can a group of AI models teach each other to reason, using each other's guesses instead of an answer key?
What mechanisms do peer predictions use to generate reward signals for training?
This explores how a group of models can replace an answer key or human labeler by using each other's predictions as the training reward, and what makes that signal usable.
This explores how a group of models can replace an answer key or human labeler by using each other's predictions as the training reward, and what makes that signal usable. The corpus has one note that addresses this directly. It describes the mechanism's outline but not its exact scoring formula.
The core move is in Can peer models replace external judges for reward signals? (Co-RL). Several separate models train side by side, and each one's reward comes from its peers' predictions, not from ground-truth labels or a trained judge. Peers are used instead of the model's own output because a model grading itself inherits its own biases and tends to collapse. The note's main finding is that diversity is what makes this work. Mixed cohorts beat a single model rewarding itself, improve reasoning across benchmarks, and often come close to training on real labels.
Peer prediction is one of several ways to remove the trained reward model. Can language models replace reward models with internal signals? sorts late-2025 work into three substitutions. Pairwise self-judgment replaces the reward model, internal belief-shift replaces the critic, and self-distillation from rich feedback replaces the explicit reward. The belief-shift version, Can an agent's own beliefs guide credit assignment without critics?, is a useful contrast. Its reward is the change in the agent's own probability estimate from turn to turn, so credit is dense and needs no critic. All three of those signals come from inside a single policy. The peer approach takes its signal from outside the model, but from models of the same kind.
Two notes suggest where peer rewards could go wrong. Does reward hacking always stem from the same failure? argues that reward hacking always has one root: optimizing against a signal that only partly represents the real task. A peer-derived reward is still a proxy in that sense. A diverse cohort is a plausible defense, since different models are unlikely to share the same blind spots, but that is my inference, not something the corpus tests. Can scalar rewards capture all the information in agent feedback? adds a second limit. A scalar reward says how well an action did but not how to change it, so a peer score would share that limitation.
The corpus doesn't say how peer predictions are turned into a number, such as whether that means agreement, ranking or something else. If you want that level of detail, the Co-RL paper itself is the next stop.
Sources 5 notes
Co-RL trains decoupled models using peer predictions as rewards, avoiding the bias and collapse of self-generated feedback. Heterogeneous cohorts consistently improve reasoning across benchmarks and often match ground-truth supervised training.
Late-2025 RL literature independently converges on three patterns that replace different RLHF components: pairwise self-judgment replaces the reward model, internal belief-shift replaces the critic, and rich-feedback self-distillation replaces explicit reward signals. Each emerges from the policy's own computations, making the trained reward classifier optional.
ΔBelief-RL uses log-ratios of sequential probability estimates to assign per-turn credit without critic networks or process reward models. Tested on 20 Questions, smaller models trained this way matched or exceeded prior SOTA and larger baselines while generalizing beyond training.
Reward hacking arises during weight training, output selection, and prompt revision through a shared failure: optimization against signals that incompletely represent the actual task. The substrate matters less than the misalignment between the scoring function and ground truth.
Natural feedback carries two orthogonal types of information: evaluative (how well an action performed) and directive (how it should change). Scalar rewards capture evaluation but discard directional specifics that token-level distillation can recover, making the two complementary rather than redundant.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Reward Reasoning Model
- Co-RL: Unsupervised Reasoning Emerges from Diverse Cohort in Multi-agent RL
- Learning to Reason without External Rewards
- Intrinsic Credit Assignment for Long Horizon Interaction
- Can Large Reasoning Models Self-Train?
- Reinforcement Learning via Self-Distillation
- Monitoring and Discovering Reward Hacking with Internal Representations during LLM Evaluations
- Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts