Can peer models replace external judges for reward signals?

The paper's claim is that "unsupervised reasoning can emerge through cooperative multi-agent training." CO-RL optimizes multiple decoupled models, "sharing no parameters," simultaneously, with each model's reward derived from its peers' predictions rather than from ground-truth labels or an external judge. The abstract reports that this consistently improves diverse LLMs and VLMs across text-only and multimodal reasoning benchmarks, outperforming prior self-rewarding and multi-agent RL approaches and, "in many settings," matching or surpassing training with ground-truth rewards.

The framing starts from a limit of self-rewarding RL. The introduction lists the single-model signals in use: agreement with a majority-vote prediction, self-certainty, predictive entropy, and consistency across paraphrased inputs or moving-average policies. It then observes that "these signals remain within a single model's own predictions." Training only on self-generated feedback can reinforce existing biases, reduce response diversity, and end in homogenized responses and training collapse. The paper's mechanism claim is that heterogeneous cohorts (different model families, different sizes, rephrased training samples) reduce "the correlated errors that drive self-reinforcing feedback loops." A separate theoretical analysis is said to show that cross-agent supervision "expands the set of initial conditions that converge to the correct solution," so errors a lone self-rewarding model would lock in can be corrected by a peer.

Set against Can language models replace reward models with internal signals?, CO-RL adds a different axis. Those three patterns each swap out one component of the RLHF/RLVR stack but keep the reward source inside one policy. CO-RL moves the reward source outside the policy without bringing back a verifier or a trained judge, so the substitution is about who supplies the signal rather than which component is replaced. It also names the failure that in-policy signals share. The self-certainty style of reward in Can model confidence alone replace external answer verification? is one of the single-model signals the paper's introduction is positioned against. By comparison, Can adversarial critics replace task-specific verifiers for reasoning? replaces the verifier with an adversarial critic against expert data, whereas CO-RL needs neither demonstrations nor a critic, only other learners.

The excerpt leaves a lot open. It does not say how peer predictions become a reward (agreement, ranking, or something else), how many agents were used, which benchmarks or base models, or how large the gains and the ground-truth-matching cases were. The word "consistently" for the diversity effect is the authors' summary and cannot be checked here. The paper itself names the open question: how "the number, diversity, and interaction topology of agents shape cross-agent learning," and how adaptive supervision might better exploit complementary expertise. What the excerpt supports is a narrower claim than "peers beat self-rewarding everywhere": in the settings tested, heterogeneity across the cohort is the stated reason label-free training stays stable and improves, and any reading of self-rewarding collapse as a single-model problem should account for it.