VRBench：长叙事视频多步推理基准测试

摘要

我们推出VRBench，这是首个专为评估大模型多步推理能力而设计的长篇叙事视频基准，旨在解决现有评估方法忽视时间推理和程序有效性的局限。该基准包含1,010段长视频（平均时长1.6小时），以及9,468个人工标注的多步问答对和30,292个带时间戳的推理步骤。这些视频通过多阶段筛选流程精心挑选，包括专家互评，以确保情节连贯性。我们开发了一个人机协作框架，用于生成连贯的推理链，每条链均需包含多个基于时间线索的步骤，涵盖七种类型（如事件归因、隐含推理）。VRBench设计了一个多阶段评估流程，从结果和过程两个层面评估模型性能。除针对最终结果的多项选择题外，我们还提出了一种基于进展水平的LLM引导评分指标，从多个维度全面评估推理链的质量。通过对12个LLM和16个VLM在VRBench上的广泛评估，我们进行了深入分析，并提供了推动多步推理领域发展的宝贵见解。

English

We present VRBench, the first long narrative video benchmark crafted for evaluating large models' multi-step reasoning capabilities, addressing limitations in existing evaluations that overlook temporal reasoning and procedural validity. It comprises 1,010 long videos (with an average duration of 1.6 hours), along with 9,468 human-labeled multi-step question-answering pairs and 30,292 reasoning steps with timestamps. These videos are curated via a multi-stage filtering process including expert inter-rater reviewing to prioritize plot coherence. We develop a human-AI collaborative framework that generates coherent reasoning chains, each requiring multiple temporally grounded steps, spanning seven types (e.g., event attribution, implicit inference). VRBench designs a multi-phase evaluation pipeline that assesses models at both the outcome and process levels. Apart from the MCQs for the final results, we propose a progress-level LLM-guided scoring metric to evaluate the quality of the reasoning chain from multiple dimensions comprehensively. Through extensive evaluations of 12 LLMs and 16 VLMs on VRBench, we undertake a thorough analysis and provide valuable insights that advance the field of multi-step reasoning.

VRBench：长叙事视频多步推理基准测试

VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

摘要

Support