從RLVR到RLSVR：任務轉換誘發自驗證獎勵，以實現開放式大語言模型自我改進

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

July 26, 2026

作者: Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao

cs.AI

摘要

基於可驗證獎勵的強化學習（RLVR）已透過大規模優化推動了面向推理的大型語言模型（LLM）的近期進展。然而，其適用性在很大程度上仍局限於數學與程式編寫等可對正確性進行確定性驗證的領域。反之，開放式任務通常依賴人類偏好、獎勵模型或基於LLM的評判器，因而引入評估偏差、評判器能力瓶頸以及額外的推理成本。 借鑑自監督學習中建構前置任務以從數據本身獲取監督信號的原則，我們提出「基於自我可驗證獎勵的強化學習」（RLSVR），一種基於任務轉換的訓練範式，旨在將RLVR擴展至開放式任務。RLSVR將開放式任務轉換為可驗證的代理環境，其內部規則與互動結果可自動產生獎勵信號。我們以SpyRL實例化RLSVR；SpyRL是一個受「誰是臥底？」啟發的多智能體自博弈環境。智能體接收不對稱資訊，完成相同的目標任務，並投票指認指定的臥底。由於臥底身份預先確定，投票結果提供了完全可驗證的獎勵，同時成功指認仍與輸出品質密切相關。在文本摘要、創意寫作與數學推理上的實驗顯示，SpyRL在不可驗證任務上優於現有的自我改進方法，並在可驗證的推理任務上取得一致的增益。這些結果表明，任務轉換能將可擴展的、基於RLVR的自我改進擴展至本質上可驗證領域之外。模型與程式碼已於 https://github.com/wangqinsi1/SpyRL 發布。

English

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verified. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs.Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a multi-agent self-play environment inspired by Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/SpyRL.