VRBench：長篇敘事視頻中多步推理的基準測試

摘要

我們推出VRBench，這是首個專為評估大型模型多步推理能力而設計的長篇敘事視頻基準，旨在解決現有評估中忽視時間推理與程序有效性的局限。該基準包含1,010部長視頻（平均時長1.6小時），以及9,468組人工標註的多步問答對和30,292個帶有時間戳的推理步驟。這些視頻通過多階段篩選流程精心挑選，包括專家評審，以確保情節連貫性。我們開發了一個人機協作框架，用於生成連貫的推理鏈，每條鏈均需多個基於時間的步驟，涵蓋七種類型（如事件歸因、隱含推斷）。VRBench設計了一個多階段評估管道，從結果和過程兩個層面對模型進行評估。除了針對最終結果的多項選擇題外，我們還提出了一種基於進展層次的大語言模型引導評分指標，從多個維度全面評估推理鏈的質量。通過對12種大語言模型和16種視覺語言模型在VRBench上的廣泛評估，我們進行了深入分析，並提供了推動多步推理領域發展的寶貴見解。

English

We present VRBench, the first long narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities, addressing limitations in existing evaluations that overlook temporal reasoning and procedural validity. It comprises 1,010 long videos (with an average duration of 1.6 hours), along with 9,468 human-labeled multi-step question-answering pairs and 30,292 reasoning steps with timestamps. These videos are curated via a multi-stage filtering process including expert inter-rater reviewing to prioritize plot coherence. We develop a human-AI collaborative framework that generates coherent reasoning chains, each requiring multiple temporally grounded steps, spanning seven types (e.g., event attribution, implicit inference). VRBench designs a multi-phase evaluation pipeline that assesses models at both the outcome and process levels. Apart from the MCQs for the final results, we propose a progress-level LLM-guided scoring metric to evaluate the quality of the reasoning chain from multiple dimensions comprehensively. Through extensive evaluations of 12 LLMs and 16 VLMs on VRBench, we undertake a thorough analysis and provide valuable insights that advance the field of multi-step reasoning.

VRBench：長篇敘事視頻中多步推理的基準測試

VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

摘要

Support