Can environment structure replace external judges in RL?

The paper proposes Reinforcement Learning with Self-Verifiable Rewards (RLSVR) as a way to extend RLVR beyond mathematics and coding. Its claim is that open-ended tasks do not need an external judge if the task itself is transformed: "instead of approximating an unverifiable quality objective with external judges or reward models, RLSVR transforms the task into a proxy environment whose environment-assigned latent variables make rewards verifiable by construction." The abstract names what this avoids: the "evaluation bias, judge capability bottlenecks, and additional inference costs" that come with human preferences, reward models, and LLM-based judges.

The paper's instantiation is SpyRL, a self-play game inspired by Who Is the Spy? Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, the environment already holds the answer, and "output-quality assessment" becomes "a verifiable identity-recognition problem." The reward comes from the interaction outcome, meaning the votes, and not from anyone's opinion of the outputs. The borrowed precedent is self-supervised learning, where pretext tasks derive supervision "from the data itself." RLSVR applies that move to the reward side of RL.

The nearest existing notes remove the external verifier by other routes. Can model confidence alone replace external answer verification? reads the reward off the model's own probabilities. Can models learn to judge themselves without external rewards? keeps a judge but has the model play it. RLSVR is a different move because the reward source is environment structure, not the model's belief and not the model's judgment. It is closest in spirit to Can reasoning during evaluation reduce judgment bias in LLM judges?, which also converts an unverifiable task into one with a checkable answer. J1 builds that answer through synthetic high and low quality response pairs, while RLSVR builds it through a hidden variable the game assigns. That would add a fourth entry to the substitutable patterns in Can language models replace reward models with internal signals?, one that replaces the reward channel with a designed environment.

The excerpt reports outcomes, not mechanics. SpyRL is said to consistently outperform existing self-improvement methods on summarization and creative writing, to yield "further gains even on already-verifiable" mathematical reasoning tasks, and to produce vote-based rewards that "align closely with human and LLM quality judgments." It does not give the models, baselines, effect sizes, number of agents, or how the information asymmetry is built. It also does not explain why identifying the spy should track output quality, or whether a policy could raise its vote reward by making outputs distinguishable in ways unrelated to quality. The alignment claim is stated without a measure. What the excerpt supports is the design principle: a reward that is verifiable by construction can be engineered into a task through a latent variable the environment controls. Whether SpyRL's particular game keeps that reward tied to quality has to be checked in the full paper.