基于视觉推理的强化学习

Grounded Reinforcement Learning for Visual Reasoning

May 29, 2025

作者: Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J. Tarr, Aviral Kumar, Katerina Fragkiadaki

cs.AI

摘要

尽管通过思维链进行强化学习（RL）已显著提升了语言模型在数学和编码等任务中的表现，但视觉推理因其要求模型引导视觉注意力、解析感知输入并将抽象推理基于空间证据而引入了额外的复杂性。我们提出了ViGoRL（视觉基础强化学习），这是一种通过RL训练的视觉-语言模型，旨在明确地将每一步推理锚定到特定的视觉坐标上。受人类视觉决策启发，ViGoRL学习生成空间基础推理轨迹，在每一步引导视觉注意力至任务相关区域。当需要精细探索时，我们新颖的多轮RL框架使模型能够在推理过程中动态放大预测坐标。在一系列视觉推理基准测试中——包括用于空间推理的SAT-2和BLINK、用于视觉搜索的V*bench，以及用于网页基础推理的ScreenSpot和VisualWebArena——ViGoRL始终优于缺乏明确基础机制的有监督微调和传统RL基线。结合多轮RL与放大视觉反馈显著提升了ViGoRL在定位小型GUI元素和视觉搜索任务上的表现，在V*Bench上达到了86.4%的准确率。此外，我们发现基础推理增强了其他视觉行为，如区域探索、基础子目标设定和视觉验证。最后，人类评估显示，模型的视觉参考不仅空间定位准确，而且有助于理解模型的推理步骤。我们的结果表明，视觉基础强化学习是赋予模型通用视觉推理能力的强大范式。

English

While reinforcement learning (RL) over chains of thought has significantly advanced language models in tasks such as mathematics and coding, visual reasoning introduces added complexity by requiring models to direct visual attention, interpret perceptual inputs, and ground abstract reasoning in spatial evidence. We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with RL to explicitly anchor each reasoning step to specific visual coordinates. Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces, guiding visual attention to task-relevant regions at each step. When fine-grained exploration is required, our novel multi-turn RL framework enables the model to dynamically zoom into predicted coordinates as reasoning unfolds. Across a diverse set of visual reasoning benchmarks--including SAT-2 and BLINK for spatial reasoning, V*bench for visual search, and ScreenSpot and VisualWebArena for web-based grounding--ViGoRL consistently outperforms both supervised fine-tuning and conventional RL baselines that lack explicit grounding mechanisms. Incorporating multi-turn RL with zoomed-in visual feedback significantly improves ViGoRL's performance on localizing small GUI elements and visual search, achieving 86.4% on V*Bench. Additionally, we find that grounding amplifies other visual behaviors such as region exploration, grounded subgoal setting, and visual verification. Finally, human evaluations show that the model's visual references are not only spatially accurate but also helpful for understanding model reasoning steps. Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.