Visionary-R1：利用強化學習緩解視覺推理中的捷徑問題

Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning

May 20, 2025

作者: Jiaer Xia, Yuhang Zang, Peng Gao, Yixuan Li, Kaiyang Zhou

cs.AI

摘要

學習通用推理能力一直是人工智慧領域中長期存在的挑戰。近期針對大型語言模型（LLMs）的研究，例如DeepSeek-R1，顯示出透過如GRPO等強化學習技術，能夠讓預訓練的LLMs利用簡單的問答對來發展推理能力。本文旨在訓練視覺語言模型（VLMs），使其能夠透過強化學習和視覺問答對來對圖像數據進行推理，而無需任何明確的思維鏈（CoT）監督。我們的研究發現，僅僅將強化學習應用於VLM——透過提示模型在提供答案前生成推理鏈——可能會導致模型從簡單問題中發展出捷徑，從而降低其在未見過數據分佈上的泛化能力。我們認為，緩解捷徑學習的關鍵在於鼓勵模型在推理前先對圖像進行解釋。因此，我們訓練模型遵循「描述-推理-回答」的輸出格式：首先生成圖像的詳細描述，接著構建深入的推理鏈。當在273K個無CoT的視覺問答對上進行訓練，並僅使用強化學習時，我們名為Visionary-R1的模型在多個視覺推理基準測試中超越了強大的多模態模型，如GPT-4o、Claude3.5-Sonnet和Gemini-1.5-Pro。

English

Learning general-purpose reasoning capabilities has long been a challenging problem in AI. Recent research in large language models (LLMs), such as DeepSeek-R1, has shown that reinforcement learning techniques like GRPO can enable pre-trained LLMs to develop reasoning capabilities using simple question-answer pairs. In this paper, we aim to train visual language models (VLMs) to perform reasoning on image data through reinforcement learning and visual question-answer pairs, without any explicit chain-of-thought (CoT) supervision. Our findings indicate that simply applying reinforcement learning to a VLM -- by prompting the model to produce a reasoning chain before providing an answer -- can lead the model to develop shortcuts from easy questions, thereby reducing its ability to generalize across unseen data distributions. We argue that the key to mitigating shortcut learning is to encourage the model to interpret images prior to reasoning. Therefore, we train the model to adhere to a caption-reason-answer output format: initially generating a detailed caption for an image, followed by constructing an extensive reasoning chain. When trained on 273K CoT-free visual question-answer pairs and using only reinforcement learning, our model, named Visionary-R1, outperforms strong multimodal models, such as GPT-4o, Claude3.5-Sonnet, and Gemini-1.5-Pro, on multiple visual reasoning benchmarks.