SYNTHESIS NOTE
Topics›Reinforcement Learning›this note

Can adversarial critics replace task-specific verifiers for reasoning?

Explores whether an adversarial game between policy and critic can substitute for explicit verifiers in RL-based reasoning training. Matters because many domains lack the task-specific validators that make current reasoning RL possible.

Synthesis note · 2026-02-22 · sourced from Reinforcement Learning

A fundamental limitation of RL for reasoning: RLVR requires task-specific verifiers (math checkers, code test suites) that don't exist for many reasoning-intensive domains. Expert demonstrations are abundant (Stack Exchange answers, domain expert explanations) but SFT on demonstrations doesn't produce the reasoning behaviors that large-scale RL training elicits. RARO bridges this gap using inverse reinforcement learning.

The mechanism is an adversarial game. A policy learns to produce expert-like answers via explicit CoT reasoning. A relativistic critic learns to discriminate between expert and policy answers via pairwise comparison. Both are trained jointly and continuously via RL, requiring careful stabilization techniques. The critic's discrimination signal serves as the reward for the policy — when the critic can't distinguish policy from expert, the policy has learned expert-level reasoning.

The results are significant: RARO outperforms strong verifier-free baselines on Countdown, DeepMath, and Poetry Writing, and enjoys the same robust scaling trends as RL with verifiers. This means the scaling properties of RLVR are not specific to verifiable rewards — they emerge from the RL training dynamics themselves, with the adversarial critic providing a sufficient substitute for ground-truth verification.

This extends the frontier of RL-for-reasoning to any domain with expert demonstrations. Since Does critiquing errors teach deeper understanding than imitating correct answers?, RARO leverages a similar mechanism — the adversarial training forces the model to develop genuine reasoning rather than surface-level imitation, because the critic can distinguish superficial pattern matching from actual expert-like problem solving.

VeriFree as a second verifier-free approach: VeriFree takes a different route to the same goal — extending R1-Zero-style RL training to domains without rule-based verifiers. Instead of an adversarial critic, VeriFree generates only the reasoning trace and concatenates it with the reference answer, then evaluates the likelihood of the reference answer conditioned on both. This likelihood serves as both a reward signal for policy gradients on the reasoning trace and a weighting term for supervised training. VeriFree is architecturally simpler than RARO (no adversarial game) and eliminates the need for even a model-based verifier, reducing compute overhead. See Can reasoning improvement work without answer verification?. The two approaches bracket the design space: RARO uses adversarial dynamics for richer signal, VeriFree uses reference-conditioned likelihood for simplicity.

Inquiring lines that read this note 41

This note is a source for these research framings, grouped by the broader line of inquiry each explores. Scan the bold lines of inquiry; follow any specific question forward.

How can verification systems keep pace with AI generation? How can evaluation systems stay reliable and resistant to AI-driven gaming? Is policy entropy collapse the central bottleneck for reasoning-focused RL scaling? How do prompt injection attacks succeed or fail across defenses? What prevalence and forms of reward hacking occur in model evaluation? What biases affect LLM judges and how can they be reduced? How do multi-agent systems balance consensus-building with error detection in collaborative reasoning? Does RLHF training optimize perceived quality over practical accuracy? Can external verification replace human reasoning oversight for correctness? How does reinforcement learning discover and optimize reasoning strategies versus supervised training? Can AI systems achieve sustainable self-improvement without external human guidance? Why do users misjudge the credibility of AI-generated and verified content? What limits reinforcement learning for reasoning's ability to expand model capability? Why does fine-tuning for specific tasks degrade broader reasoning capability? How do agents coordinate to violate constraints despite individual-level safeguards? Can aggregate reward models preserve diverse human preferences? Can AI-generated text be reliably distinguished from human writing? How should evaluation environments be designed as security boundaries?

Related concepts in this collection 6

This note in its neighbourhood — explore the map, then jump to a related concept in the list below.

Concept map
18 direct connections · 111 in 2-hop network ·medium cluster Open in graph ↗

Click a node to walk · click center to open · click Open in graph to see this note in the full knowledge graph

your link semantically near linked from elsewhere

Related papers in this collection 8

Papers most semantically related to this note, ranked by cosine similarity in the embedding space.

Original note title

inverse rl from demonstrations enables reasoning training without task-specific verifiers