AI models can be trained by arguing with each other, but does that still work when no answer key exists?
Can training-time debate work on tasks beyond mathematics and verifiable answers?
This explores whether training-time debate, where AI models challenge each other's answers while they learn, still helps when there is no answer key to check against, such as in open-ended or contested domains.
This explores whether training-time debate, where AI models challenge each other's answers while they learn, still helps when there is no answer key to check against. Nobody in this collection has shown that it does. All the positive evidence is from math, and the authors of that work name the jump beyond math as their most important unanswered question.
The math result is strong. A generator and a critic argued while a frozen, weaker judge decided who won. The judge stayed reliable throughout training, and peak validation accuracy came out 45% higher than in single-player training, which quickly learned to exploit the judge's mistakes and collapsed Can debate training prevent reward hacking by weaker judges?. The benefit appears during training itself, because each agent has to survive the other's challenges across many steps Does debate actually help during AI training, not just inference?. But the anti-hacking effect was measured only on checkable answers. Without an answer key, a critic might win by sounding convincing rather than by being right Does debate prevent reward hacking without ground truth?.
The inference-time evidence points the same way. Multi-agent debate raises accuracy on math and logic. In contested domains, without external evidence checking, it reverses, and persuasive framing beats correctness so the debate produces false consensus When does debate actually improve reasoning accuracy?. That is a different setting from training, so it suggests a risk rather than proving one. It does hint that in math the checkable answer quietly does much of the work of keeping debate honest.
The corpus also shows what people do instead when answers can't be checked. VeriFree skips the verifier and rewards the model by how likely a reference answer is given its reasoning. It matches verifier-based training on MMLU-Pro, GPQA and SuperGPQA Can reasoning improvement work without answer verification?. It still needs reference answers, so it drops the verifier but keeps the answer key. Debate would have to drop both to reach truly open-ended tasks. The stakes are real: the strongest small-model reasoning results are explicitly limited to tasks with clean reward signals Can small models match frontier reasoning without massive scale?. Reasoning training can even hurt knowledge-heavy fields like medicine Why does reasoning training help math but hurt medical tasks?. One note frames the reusable unit of reasoning training as a verifier-bearing feedback interface, not a dataset What is the actual reusable unit of reasoning data?. On that view, debate is a candidate substitute for the verifier where none exists, and the collection has not yet tested it there.
Sources 8 notes
The paper measured debate's anti-hacking benefit only on mathematics with checkable answers, and explicitly flagged transfer to ground-truth-free domains as its most critical open question. Without answer keys, critics might win through persuasion rather than accuracy.
Training-time debate prevents reward hacking and judge degradation by forcing agents to challenge each other's outputs during learning, maintaining signal quality across many training steps where single-agent baselines collapse.
On math tasks, debate between a generator and critic adjudicated by a frozen weaker judge maintained judge performance throughout training and achieved 45% higher peak validation accuracy than single-player RLAIF, which quickly exploited the judge's errors and collapsed in accuracy.
Multi-agent debate boosts accuracy on verifiable tasks like math and logic, but reverses in contested domains without external evidence checking. Without verification, persuasive framing wins over correctness, making debate a false-consensus generator rather than accuracy amplifier.
VeriFree bypasses answer verification entirely by using the conditional probability of reference answers given generated reasoning traces as both reward signal and training weight. This approach matches or surpasses verifier-based methods on MMLU-Pro, GPQA, and SuperGPQA without rule-based or model-based verifiers.
Show all 8 sources
A 3B model trained with curriculum SFT and multi-domain RL reaches 94.3 AIME26 and 80.2 LiveCodeBench scores matching much larger systems. The result is bounded to verifiable tasks with checkable ground truth, where RL can provide clean reward signals.
Two-phase inference model shows knowledge retrieval operates in lower network layers while reasoning adjustment happens in higher layers. This separation explains why reasoning training improves math but can degrade knowledge-intensive domains like medicine.
The reusable unit in post-training is a feedback interface entangled with six factors: verifier, base model, lineage, optimizer, scaffold, and budget. Changing any one alters the same data's effect, making attribution tractable only when these are jointly released.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Debate Training Reduces Reward Hacking in RLAIF
- Reinforcing General Reasoning without Verifiers
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Escaping the Verifier: Learning to Reason via Demonstrations
- Eliciting Reasoning in Language Models with Cognitive Tools
- An Enigma of Artificial Reason: Investigating the Production-Evaluation Gap in Large Reasoning Models
- Is Chain-of-Thought Reasoning of LLMs a Mirage? A Data Distribution Lens
- Debating with More Persuasive LLMs Leads to More Truthful Answers