V-Rubrics：基於評分量規強化學習的視覺忠實度

摘要

視覺語言模型能產生流暢的回答，但這些回答可能缺乏足夠的視覺證據支撐：單一個未經支持的物件、圖表數值或中間推理，就可能使原本看似合理的回答失效。我們主張這是多模態後訓練中的信用分配失敗。標量結果獎勵能指示回答是否可被接受，但無法指出哪些視覺事實有充分依據、哪些推理步驟有效、或哪些指令約束被遺漏。我們提出「基於視覺評分量規的強化學習」（Visual Rubrics-Based Reinforcement Learning），將參考回答分解為原子命題，並依照視覺忠實度（VF）、推理一致性（RC）與指令遵循（IF）來評分生成回答。所產生的評分量規項目提供結構化的部分給分，並在可取得支持性證據片段時，將評分量規信用定位至對應片段。我們首先在公開的 OpenMMReasoner-SFT-874K 語料庫上微調 Qwen3-VL-8B-Instruct，取得 SFT 檢查點，沿用 OpenMMReasoner 的冷啟動資料配方。我們建構了 V-Rubrics 50K，這是一個包含 50,248 個範例的訓練集，來源為 17 個視覺接地資料源；我們先套用規則式篩選，再根據拒絕抽樣分數推導範例難度，最後使用 Gemini-3-Pro 以相同的結構化提示與協定對每個範例進行標註。我們基於同一個 SFT 檢查點訓練模型，使用逐元件、前綴局部化的評分量規信用。實驗顯示，我們以評分量規為基礎的 GRPO 在共同的 SFT 基線與僅使用答案的 GRPO 上均有改善，在知識導向與視覺接地推理基準上提升最為顯著。結果表明，評分量規是視覺後訓練中一種有用的獎勵抽象。

English

Vision-language models can produce fluent answers that are insufficiently grounded in the visual evidence: a single unsupported object, chart value, or intermediate inference can undermine an otherwise plausible response. We argue that this is a credit-assignment failure in multimodal post-training. Scalar outcome rewards indicate whether an answer is acceptable, but do not identify which visual facts are grounded, which reasoning steps are valid, or which instruction constraints are missed. We introduce Visual Rubrics-Based Reinforcement Learning, which decomposes reference responses into atomic propositions and scores generated answers along Visual Faithfulness (VF), Reasoning Consistency (RC), and Instruction Following (IF). The resulting rubric items provide structured partial credit and localize rubric credit when supporting evidence spans are available. We first obtain an SFT checkpoint by fine-tuning Qwen3-VL-8B-Instruct on the public OpenMMReasoner-SFT-874K corpus, adapting OpenMMReasoner's cold-start data recipe. We construct V-Rubrics 50K, a 50,248-example training set from 17 visually grounded sources, by applying rule-based filters before deriving example difficulty from rejection-sampling scores and then annotating every example with Gemini-3-Pro under the same structured prompt and protocol. We train our model based on the same SFT checkpoint using component-wise, prefix-localized rubric credit. Experiments show that our rubricbased GRPO improves over both the shared SFT baseline and answer-only GRPO, with the largest gains on knowledge-oriented and visually grounded reasoning benchmarks. The results show rubrics as a useful reward abstraction for visual post-training.