以标注为轨迹展开：面向视频多模态大语言模型的高效可扩展强化学习

摘要

多模态大语言模型（MLLMs）已成为统一视频感知的主流范式。然而，在大型多任务数据集上进行后训练仍面临挑战，因为现有的强化学习方法即使采用昂贵的思维链（CoT）生成，也仅在同策略组内采样少量高质量轨迹。本文研究了视频MLLMs强化学习后训练的样本效率与可扩展性，并提出了OraRL方法。我们发现了一个被忽视的标注作用：每条标注除了用于对采样结果评分外，还可以作为一条“指导（oracle）轨迹”进入其对应的同策略组，成为直接的正面优化目标。然而，直接整合指导轨迹并非易事：高奖励的指导轨迹会抬高组内基线，从而将原本为正的策略优势变为负值——我们将这一失效模式称为“优势反转”。OraRL的核心是一个解耦优势估计器：策略采样轨迹决定无指导基线，而指导-策略差距同时调节方向增益和独立的分离式指导优势。符号均衡剪枝进一步提升了效率：仅保留指导轨迹及每个符号下最强的若干采样轨迹，OraRL的步进时间仅为SFT的2.2倍，远低于GRPO配合CoT时的4.9倍。OraRL随模型规模和数据量同步扩展，在0.8B至9B参数范围内超越其基础模型，并在多达10万条提示上优于GRPO。在不使用思维链的情况下，Video-ORA-9B解码只需130毫秒，而后者需要4,780毫秒。与各自此前的最佳模型相比，OraRL将时间mIoU从62.5提升至66.0，跟踪AO从73.0提升至78.2，分割性能从64.3提升至70.4，三项空间智能基准的宏平均值从51.0提升至56.1；在VSI-Bench上，OraRL得分为73.1，而GPT-5为55.0，Gemini-3-Pro为55.1。

English

Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.