「註釋即展開：影片多模態大語言模型的高效可擴展強化學習」

摘要

多模態大型語言模型（MLLMs）已成為統一影片感知的主流範式。然而，在大型多任務資料集上進行後期訓練仍具挑戰性，因為現有的強化學習方法即使在昂貴的思維鏈（CoT）生成下，也只能從同策略群組中採樣到少量高品質軌跡。本文研究了影片MLLMs強化學習後期訓練的樣本效率與可擴展性，並提出了OraRL。我們發現了註解一個被忽略的作用：除了對軌跡進行評分，每個註解都可以作為教師軌跡（oracle rollout）進入其同策略群組，成為直接的正向最佳化目標。然而，直接整合教師軌跡並非易事：高獎勵的教師軌跡會提高群組基線，並使原本正向的策略優勢發生反轉，我們將此失敗現象稱為「優勢反轉」（advantage inversion）。OraRL的核心是一個解耦的優勢估計器：策略軌跡決定無教師基線，而教師-策略差距則同時調節方向性增益與一個獨立的分離教師優勢。符號平衡剪枝提升了效率：透過僅保留教師軌跡及各符號中最強的軌跡，OraRL只需SFT步進時間的2.2倍，遠低於GRPO搭配CoT所需的4.9倍。OraRL隨模型規模與資料量擴展，在0.8B至9B的範圍內超越其骨幹模型，並在多達10萬個提示詞的規模下超越GRPO。無需思維鏈，Video-ORA-9B僅需130毫秒即可完成解碼，而非4,780毫秒。與先前各基準的最佳模型相比，它將時間mIoU從62.5提升至66.0，追蹤AO從73.0提升至78.2，分割從64.3提升至70.4，三個基準的空間智慧宏觀平均值從51.0提升至56.1；在VSI-Bench上，它取得73.1分，而GPT-5為55.0分、Gemini-3-Pro為55.1分。

English

Multimodal large language models (MLLMs) have become a prevailing paradigm for unified video perception. However, post-training on large multi-task datasets remains challenging, as existing reinforcement learning methods sample on-policy groups with few high-quality rollouts even with costly chain-of-thought (CoT) generation. In this paper, we study the sample efficiency and scalability of RL post-training for video MLLMs and introduce OraRL. We identify an overlooked role for annotations: Beyond scoring rollouts, each can enter its on-policy group as an oracle rollout, a direct positive optimization target. Direct oracle integration, however, is nontrivial: a high-reward oracle raises the group baseline and inverts otherwise positive policy advantages, a failure we term advantage inversion. At the core of OraRL is a decoupled advantage estimator: policy rollouts determine an oracle-free baseline, while the oracle-policy gap modulates both a directional gain and a separate detached oracle advantage. Sign-balanced pruning improves efficiency: by retaining only the oracle and the strongest rollouts of each sign, OraRL requires just 2.2x the step time of SFT, less than half the 4.9x required by GRPO with CoT. OraRL scales with model size and data, surpassing its backbone from 0.8B to 9B and GRPO up to 100k prompts. Without chain-of-thought, Video-ORA-9B decodes in 130 ms instead of 4,780 ms. Compared with the respective prior best models, it raises temporal mIoU from 62.5 to 66.0, tracking AO from 73.0 to 78.2, segmentation from 64.3 to 70.4, and the three-benchmark spatial-intelligence macro average from 51.0 to 56.1; on VSI-Bench, it scores 73.1 against 55.0 for GPT-5 and 55.1 for Gemini-3-Pro.