从RLVR到RLSVR：任务变换催生自验证奖励以支持开放式大语言模型自我改进

摘要

基于可验证奖励的强化学习（RLVR）通过支持大规模优化，推动了面向推理的大语言模型（LLM）的近期进展。然而，其适用范围仍主要局限于数学和编程等领域，在这些领域中可以确定性验证正确性。开放式任务则通常依赖人类偏好、奖励模型或基于LLM的评判者，这引入了评估偏差、评判者能力瓶颈以及额外的推理成本。借鉴自监督学习通过构造前置任务从数据本身获取监督信号的原理，我们提出基于自可验证奖励的强化学习（RLSVR），一种基于任务变换的训练范式，旨在将RLVR扩展到开放式任务。RLSVR将开放式任务转化为可验证的代理环境，其内部规则和交互结果自动生成奖励信号。我们以SpyRL实例化RLSVR，这是一种受《谁是卧底》启发的多智能体自博弈环境。智能体接收非对称信息，完成相同的目标任务，并通过投票识别指定的卧底。由于卧底身份是预先确定的，投票结果提供完全可验证的奖励，而成功识别与输出质量密切相关。在文本摘要、创意写作和数学推理上的实验表明，SpyRL在不可验证任务上优于现有自我改进方法，并在可验证推理任务上取得一致的提升。这些结果表明，任务变换可以将基于可验证RLVR的自我改进扩展到固有可验证领域之外。模型和代码已在 https://github.com/wangqinsi1/SpyRL 发布。

English

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verified. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs.Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/SpyRL.