PerceptionRubrics：將多模態評估校準至人類感知

摘要

我們提出了 PerceptionRubrics，一個基於評分標準的評估框架，旨在解決飽和基準分數與真實世界脆弱性之間的差距。該框架將評估從整體語義比對轉向嚴格的原子級審查，將 1,038 張資訊密集的圖像與超過 12,000 個特定實例的評分標準配對。這些標準是透過創新的「循環同儕審查共識流程」建構出黃金標題後推導而成，並被提煉為雙流系統：「必須正確」 (基本事實) 和「容易錯誤」 (細粒度細節) 的評分標準。關鍵在於，PerceptionRubrics 實作了「門控評分機制」：不同於線性平均，若在強制性視覺事實（mandatory visual facts）上失敗，會觸發嚴格的二元懲罰。廣泛的評估得出幾項關鍵見解：(1) 可靠性差距：模型往往能正確驗證零散元素，卻在嚴格的合取約束（conjunctive constraints）下失敗，暴露了密集領域的脆弱性；(2) 開源-封閉分層：與推理趨勢相反，我們揭示了開源與專有前沿之間存在持續 8% 的感知差距；(3) 人類對齊的嚴謹性：我們的門控指標在對齊程度上顯著優於傳統基準，驗證了嚴格的感知保真度（perceptual fidelity）是可靠生成的前提。

English

We introduce PerceptionRubrics, a rubric-based evaluation framework that addresses the gap between saturated benchmark scores and real-world brittleness. Shifting evaluation from holistic semantic matching to rigorous atomic auditing, PerceptionRubrics pairs 1,038 information-dense images with over 12,000 instance-specific rubrics. These criteria are derived from golden captions constructed via a novel Circular Peer-Review consensus pipeline and then distilled into a dual-stream system of Must-Right (essential facts) and Easy-Wrong (fine-grained details) rubrics. Crucially, PerceptionRubrics implements a Gated Scoring mechanism: unlike linear averages, failure on mandatory visual facts triggers sharp binary penalties. Extensive evaluation yields critical insights: (1) The Reliability Gap: models often verify fragmented elements correctly yet fail strict conjunctive constraints, exposing brittleness in dense domains; (2) Open-Closed Stratification: contrary to reasoning trends, we reveal a persistent 8% perception deficit between open-source and proprietary frontiers; and (3) Human-Aligned Rigor: our gated metrics substantially out-align conventional benchmarks, validating that strict perceptual fidelity is the prerequisite for reliable generation.