通過R1-Zero式訓練提升視覺空間推理能力

Improved Visual-Spatial Reasoning via R1-Zero-Like Training

April 1, 2025

作者: Zhenyi Liao, Qingsong Xie, Yanhao Zhang, Zijian Kong, Haonan Lu, Zhenyu Yang, Zhijie Deng

cs.AI

摘要

提升多模态大语言模型（MLLMs）的推理能力日益受到关注。作为在物理世界中运作的AI代理的基石，基于视频的视觉空间智能（VSI）成为MLLMs最为关键的推理能力之一。本研究首次深入探讨了通过类似R1-Zero的训练方法来增强MLLMs的视觉空间推理能力。技术上，我们首先发现中小型Qwen2-VL模型的视觉空间推理能力无法通过思维链（CoT）提示激活。随后，我们借鉴DeepSeek-R1-Zero的方法，采用精心策划的VSI-100k数据集，引入GRPO训练以提升视觉空间推理。在研究中，我们认识到即使在GRPO中保持KL惩罚（即使数值较小）也是必要的。仅用120 GPU小时，我们的vsGRPO-2B模型，基于Qwen2-VL-2B微调，便超越了基础模型12.1%，并超越了GPT-4o。此外，我们的vsGRPO-7B模型，基于Qwen2-VL-7B微调，其性能可与最佳开源模型LLaVA-NeXT-Video-72B相媲美。同时，我们将vsGRPO与监督微调和直接偏好优化基线进行比较，观察到显著的性能优势。代码和数据集即将公开。

English

Increasing attention has been placed on improving the reasoning capacities of multi-modal large language models (MLLMs). As the cornerstone for AI agents that function in the physical realm, video-based visual-spatial intelligence (VSI) emerges as one of the most pivotal reasoning capabilities of MLLMs. This work conducts a first, in-depth study on improving the visual-spatial reasoning of MLLMs via R1-Zero-like training. Technically, we first identify that the visual-spatial reasoning capacities of small- to medium-sized Qwen2-VL models cannot be activated via Chain of Thought (CoT) prompts. We then incorporate GRPO training for improved visual-spatial reasoning, using the carefully curated VSI-100k dataset, following DeepSeek-R1-Zero. During the investigation, we identify the necessity to keep the KL penalty (even with a small value) in GRPO. With just 120 GPU hours, our vsGRPO-2B model, fine-tuned from Qwen2-VL-2B, can outperform the base model by 12.1% and surpass GPT-4o. Moreover, our vsGRPO-7B model, fine-tuned from Qwen2-VL-7B, achieves performance comparable to that of the best open-source model LLaVA-NeXT-Video-72B. Additionally, we compare vsGRPO to supervised fine-tuning and direct preference optimization baselines and observe strong performance superiority. The code and dataset will be available soon.