Does debate prevent reward hacking without ground truth?

The paper says so directly: "Several important questions remain open... The most critical is whether debate's benefits transfer to domains without verifiable ground truth." The result in Can debate training prevent reward hacking by weaker judges? is on mathematics, chosen so that hacking could be measured against checkable final answers. RLAIF's stated purpose is the opposite regime, tasks "without ground-truth labels," so the question is whether the remedy holds where it is needed.

Two readings, both open.

It transfers. The mechanism is that a critic exposes the flaw a weak judge would miss, so exploiting the judge costs the generator. Nothing in that argument obviously depends on the answer being checkable, only on the critic being able to find and state the flaw and the judge being able to recognize a stated flaw.

It does not. The vault's evidence on debate in contested domains is that persuasive framing can substitute for evidence (When does debate actually improve reasoning accuracy?) and that LLM judges systematically favor LLM-authored arguments (Do LLM judges systematically favor arguments from other LLMs?). In a domain with no answer key, a critic's most effective move could be the more persuasive attack rather than the more accurate one, and a weak judge could reward it. That would make the adversary a new exploit surface rather than a guard. This is a vault reading, not something the paper argues.

A second reason for doubt comes from the selection argument in Does iterative training against detected failures prevent actual compliance?: a detector inside a training loop rewards the policies it fails to detect. The critic is a detector placed in the loop by design, and the generator's reward runs through the judge's adjudication of what the critic exposes. Where flaws can be checked, passing the critic and being right plausibly track each other; where nothing can be checked they may come apart. The argument was made for detectors that penalize what they detect, so whether an adversary that is itself trained escapes it is open, and the excerpt does not say how the critic's role is trained. This too is a vault reading, not the paper's.

What a test would need: a domain where ground truth exists for evaluation only, hidden from both the training loop and the judge, so hacking stays measurable while the training regime looks unlabeled. The vault already holds the two pieces in other settings: Can deterministic checks protect LLM judges from failure? keeps a partition nothing that proposes changes can see, and How often do agents exploit optional shortcuts in benchmarks? measures hacking as a gap between a public score and a hidden test set. Neither was run as a training comparison of debate against single-player RLAIF. Whether that closes the gap is itself uncertain, since a domain with a hidden answer key may not have the contested character that makes the unlabeled case hard. The excerpt describes no such experiment and reports one policy-judge pair.