SVR-R1：强化学习中的自我验证引导多模态推理自举

摘要

我们提出了自验证推理器（Self-Verified Reasoner, SVR-R1），这是一个多轮强化学习框架，能将模型自身的验证转化为多模态推理的学习信号。对于每个查询，模型使用相同的权重生成答案，并输出一个二元自验证结论（是/否）。若结果为“否”，则触发第二轮修正思考；若结果为“是”或达到轮次上限，则最终确定输出并用于计算基于结果的奖励。SVR-R1 基于 GRPO 和异步多轮展开框架实现，无需外部监督或辅助评估器。我们在视觉-语言推理基准上评估了 SVR-R1，结果表明其准确率较强大的标准 GRPO 基线有大幅提升。训练动态显示，对验证的依赖逐渐降低——验证轮次减少，但测试准确率反而提高——这表明随着策略通过我们的框架内化自我修正机制并选择最自信的答案，验证与生成之间的差距逐渐缩小。SVR-R1 连接了视觉语言模型在推理时自我优化与强化学习训练这一较少探索的交叉领域，为引导多模态推理提供了一种简单而有效的方案。我们将开源 SVR-R1，以促进视觉语言模型的未来研究。

English

We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning. For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No). A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward. SVR-R1 is implemented with GRPO and an asynchronous multi-turn rollout framework and needs no external supervision or auxiliary critics. We evaluate SVR-R1 on vision-language reasoning benchmarks and show that it improves accuracy by a large margin over strong standard GRPO baselines. Training dynamics show decreasing reliance on verification-fewer verification turns, yet higher test accuracy-indicating that the gap between verification and generation narrows as the policy internalizes self-correction and chooses the most confident answer via our framework. SVR-R1 bridges the less explored intersection of inference-time self-refinement and RL training for VLMs, offering a simple yet effective recipe for bootstrapping multimodal reasoning. We will open-source SVR-R1 to facilitate future research in VLMs.