基於證據的影片問答

摘要

当前视频大语言模型（Video LLMs）在问答（QA）任务中表现出色，但大多作为黑箱运行，仅输出文本答案而缺乏可验证的视觉归因。现有的可解释性方法依赖于文本解释或稀疏边界框，难以捕捉诸如遮挡与非刚性形变等复杂视频动态。我们提出证据支持的视频问答（E-VQA），这一新任务要求模型同时输出语义答案与精确的时空证据：时间片段及密集的、带有跟踪目标分割掩码片段。为此，我们引入ST-Evidence，这是首个经人工验证的、同时涵盖判别式与生成式像素级定位的基准。对当前最优模型的评估揭示了问答准确率与真实视觉感知之间存在关键性解耦，单纯依靠规模扩展无法弥合这一差距。为解决该问题，我们开发可扩展的自动化生成流程，构建ST-Evidence-Instruct——一个16万规模的数据集，桥接高层推理与细粒度定位。基于此数据对具备视觉定位能力的视频大语言模型进行微调，使得对应尺寸的UniPixel基线模型获得显著提升（例如，7B模型在t-mean上提升+27.2，在J&F上提升+13.8），为可解释、证据支持的视频理解建立了稳健基线。代码与数据开源于 https://github.com/SalesforceAIResearch/EVQA。

English

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.