ViSurf：面向大规模视觉-语言模型的视觉监督与强化微调框架

摘要

大型视觉语言模型（LVLMs）的典型后训练范式包括监督微调（SFT）和基于可验证奖励的强化学习（RLVR）。SFT借助外部指导注入新知识，而RLVR则利用内部强化提升推理能力和整体性能。然而，我们的分析表明，SFT往往导致次优表现，而RLVR在处理超出模型内部知识库的任务时存在困难。为应对这些局限，我们提出了ViSurf（视觉监督与强化微调），一种统一的后训练范式，将SFT和RLVR的优势整合于单一阶段。我们通过分析SFT和RLVR目标的推导，确立了ViSurf目标，为这两种范式提供了统一视角。ViSurf的核心在于将真实标签注入RLVR的探索过程中，从而同时提供外部监督和内部强化。此外，我们引入了三种新颖的奖励控制策略，以稳定并优化训练过程。在多个多样化基准上的广泛实验验证了ViSurf的有效性，其表现优于单独的SFT、RLVR以及两阶段的SFT→RLVR。深入分析进一步支持了这些发现，证实了ViSurf的推导与设计原则。

English

Typical post-training paradigms for Large Vision-and-Language Models (LVLMs) include Supervised Fine-Tuning (SFT) and Reinforcement Learning with Verifiable Rewards (RLVR). SFT leverages external guidance to inject new knowledge, whereas RLVR utilizes internal reinforcement to enhance reasoning capabilities and overall performance. However, our analysis reveals that SFT often leads to sub-optimal performance, while RLVR struggles with tasks that exceed the model's internal knowledge base. To address these limitations, we propose ViSurf (Visual Supervised-and-Reinforcement Fine-Tuning), a unified post-training paradigm that integrates the strengths of both SFT and RLVR within a single stage. We analyze the derivation of the SFT and RLVR objectives to establish the ViSurf objective, providing a unified perspective on these two paradigms. The core of ViSurf involves injecting ground-truth labels into the RLVR rollouts, thereby providing simultaneous external supervision and internal reinforcement. Furthermore, we introduce three novel reward control strategies to stabilize and optimize the training process. Extensive experiments across several diverse benchmarks demonstrate the effectiveness of ViSurf, outperforming both individual SFT, RLVR, and two-stage SFT \textrightarrow RLVR. In-depth analysis corroborates these findings, validating the derivation and design principles of ViSurf.

ViSurf：面向大规模视觉-语言模型的视觉监督与强化微调框架

ViSurf: Visual Supervised-and-Reinforcement Fine-Tuning for Large Vision-and-Language Models

摘要

Support