SVR-R1：在強化學習中藉由自我驗證促進多模態推理的自舉方法

摘要

我們提出自我驗證推理器（Self-Verified Reasoner, SVR-R1），這是一種多輪強化學習框架，能將模型自身的驗證轉化為多模態推理的學習訊號。針對每個查詢，模型使用相同權重提出答案，並發布二元自我裁決（是／否）。「否」會觸發第二次機會的重新思考；「是」或達到輪次上限則輸出最終結果，用以計算基於結果的獎勵。SVR-R1 以 GRPO 和非同步多輪展開框架實現，無需外部監督或輔助評估器。我們在視覺語言推理基準上評估 SVR-R1，結果顯示其在準確率上大幅超越強大的標準 GRPO 基準。訓練動態顯示對驗證的依賴性降低——驗證輪數減少，但測試準確率卻更高——這表明隨著策略內化自我修正，並透過我們的框架選擇最有信心的答案，驗證與生成之間的差距逐漸縮小。SVR-R1 橋接了推理時自我優化與 VLM 強化學習訓練中較少被探索的交叉領域，為自舉多模態推理提供了一個簡單而有效的方案。我們將開源 SVR-R1，以促進 VLM 領域的未來研究。

English

We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source SVR-R1 to facilitate future research in VLMs.