From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verifiable. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs. Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformationbased training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy.

Introduction. Reinforcement Learning with Verifiable Rewards (RLVR) has enabled scalable training for reasoning models like OpenAI o1 (OpenAI, 2024) and DeepSeek-R1 (DeepSeek-AI, 2025). However, while RLVR excels in deterministic domains like math and coding, it remains brittle in open-ended tasks requiring subjective judgment. To bridge this gap, prior research relaxes strict verifiability using learned preference signals (e.g., RLHF (Ouyang et al., 2022), DPO (Rafailov et al., 2023)) or model-based feedback proxies, such as LLM-as-a- Judge (Zheng et al., 2023) and self-rewarding mechanisms (Yuan et al., 2024). These methods broaden the applicability of RL, but they also introduce evaluation bias, bottlenecks in judge capability, and additional inference costs. Self-supervised learning offers a useful methodological precedent for addressing this challenge. In the absence of human annotations, it constructs pretext objectives whose supervisory signals are derived automatically from the data itself (Doersch et al., 2015; Noroozi & Favaro, 2016).

Discussion / Conclusion. We propose RLSVR, a training paradigm that extends RLVR to open-ended tasks by bringing the task-transformation principle of self-supervised learning into reinforcement learning: instead of approximating an unverifiable quality objective with external judges or reward models, RLSVR transforms the task into a proxy environment whose environment-assigned latent variables make rewards verifiable by construction. We instantiate this paradigm with SpyRL, an information-asymmetric self-play game in which a predetermined spy identity turns output-quality assessment into a verifiable identity-recognition problem. Across summarization, creative writing, and mathematical reasoning, SpyRL consistently outperforms existing self-improvement methods on open-ended tasks and yields further gains even on already-verifiable ones, with vote-based rewards shown to align closely with human and LLM quality judgments.