基于证据的视频问答

摘要

当前视频大语言模型（Video LLMs）在问答任务中表现优异，但大多以黑箱方式运作，仅输出文本答案而缺乏可验证的视觉依据。现有的可解释性方法依赖于文本解释或稀疏的边界框，难以捕捉复杂的视频动态信息（如遮挡与非刚性变形）。为此，我们提出基于证据的视频问答（E-VQA）新任务，要求模型同时输出语义答案与精确的时空证据：即时间片段与密集、可追踪的对象分割掩码片段。为支持该任务，我们构建了ST-Evidence——首个经人工验证的基准数据集，用于判别式与生成式像素级定位。对现有最先进模型的评估揭示：问答准确率与真实视觉感知之间存在关键解耦，且仅靠规模扩展无法弥合这一差距。为解决此问题，我们开发了可扩展的自动化生成流程，构建了ST-Evidence-Instruct数据集（16万规模），将高层推理与细粒度定位相衔接。在此数据上微调具备空间定位能力的视频大语言模型，相较于同等规模的UniPixel基线模型获得了显著提升（例如7B模型上t-均值提升27.2，J&F指标提升13.8），为可解释、基于证据的视频理解建立了稳健基线。代码与数据已开源：https://github.com/SalesforceAIResearch/EVQA。

English

Current Video Large Language Models (Video LLMs) excel in question answering (QA) but largely operate as black boxes, providing textual answers without verifiable visual grounding. Existing explainability efforts rely on textual rationales or sparse bounding boxes, which struggle to capture complex video dynamics such as occlusions and non-rigid deformations. We propose Evidence-Backed Video Question Answering (E-VQA), a novel task requiring models to jointly output a semantic answer and precise spatio-temporal evidence: temporal segments and dense, tracked object segmentation masklets. To support this, we introduce ST-Evidence, the first human-verified benchmark for both discriminative and generative pixel-level grounding. Evaluations of state-of-the-art models reveal a critical decoupling between QA accuracy and true visual perception that scaling alone fails to bridge. To address this, we develop scalable, automated generation pipelines to create ST-Evidence-Instruct, a 160k-scale dataset bridging high-level reasoning with fine-grained grounding. Fine-tuning grounded Video LLMs on this data yields substantial gains over the corresponding size-matched UniPixel baselines (e.g., +27.2 t-mean and +13.8 J&F on a 7B model), establishing a robust baseline for explainable, evidence-backed video understanding. Code and data are available at https://github.com/SalesforceAIResearch/EVQA.