强化学习使多模态大语言模型的视觉理解能力超越了监督微调。
RL makes MLLMs see better than SFT
October 18, 2025
作者: Junha Song, Sangdoo Yun, Dongyoon Han, Jaegul Choo, Byeongho Heo
cs.AI
摘要
在多模态语言模型（MLLM）研究中，一个主流假设认为其性能主要继承自大规模参数和卓越能力的LLM主干。这一观点导致了对视觉编码器理解的空白，而视觉编码器决定了MLLM如何感知图像。近期，MLLM训练范式从监督微调（SFT）向强化学习（RL）的转变，放大了这一忽视——即对这类训练如何重塑视觉编码器及MLLM本身的分析严重不足。针对此问题，我们首先探究了训练策略对MLLM的影响，发现RL在强视觉相关的VQA基准测试中明显优于SFT。受此启发，我们通过一系列多样且深入的实验，从ImageNet分类与分割到梯度可视化，对MLLM的视觉编码器进行了关键却鲜有探索的分析。我们的结果表明，MLLM的训练后策略（即SFT或RL）不仅导致下游任务结果显著不同，还从根本上重塑了MLLM的底层视觉表征。具体而言，我们研究的关键发现是，与SFT相比，RL能生成更强且定位更精确的视觉表征，从而提升了MLLM视觉编码器的能力。随后，我们将这些发现提炼为构建强大MLLM视觉编码器的简单方案——偏好指导视觉优化（PIVOT）。当PIVOT训练的视觉编码器集成到MLLM中时，尽管所需计算成本不到标准视觉预训练的1%，其表现却超越了更大规模、训练更充分的同类模型。这一结果为推进MLLM视觉主干的发展开辟了一条高效且有效的路径。项目页面详见https://june-page.github.io/pivot/。
English
A dominant assumption in Multimodal Language Model (MLLM) research is that
its performance is largely inherited from the LLM backbone, given its immense
parameter scale and remarkable capabilities. This has created a void in the
understanding of the vision encoder, which determines how MLLMs perceive
images. The recent shift in MLLM training paradigms, from Supervised Finetuning
(SFT) to Reinforcement Learning (RL), magnifies this oversight-namely, the
significant lack of analysis on how such training reshapes the vision encoder
as well as the MLLM. To address this, we first investigate the impact of
training strategies on MLLMs, where RL shows a clear advantage over SFT in
strongly vision-related VQA benchmarks. Motivated by this, we conduct a
critical yet under-explored analysis of the vision encoder of MLLMs through
diverse and in-depth experiments, ranging from ImageNet classification and
segmentation to gradient visualization. Our results demonstrate that MLLM's
post-training strategy (i.e., SFT or RL) not only leads to distinct outcomes on
MLLM downstream tasks, but also fundamentally reshapes MLLM's underlying visual
representations. Specifically, the key finding of our study is that RL produces
stronger and precisely localized visual representations compared to SFT,
boosting the ability of the vision encoder for MLLM. We then reframe our
findings into a simple recipe for building strong vision encoders for MLLMs,
Preference-Instructed Vision OpTimization (PIVOT). When integrated into MLLMs,
a PIVOT-trained vision encoder outperforms even larger and more heavily-trained
counterparts, despite requiring less than 1% of the computational cost of
standard vision pretraining. This result opens an effective and efficient path
for advancing the vision backbones of MLLMs. Project page available at
https://june-page.github.io/pivot/