面向时序定位的因子化视频-语言模型学习

Factorized Learning for Temporally Grounded Video-Language Models

December 30, 2025

作者: Wenzheng Zeng, Difei Gao, Mike Zheng Shou, Hwee Tou Ng

cs.AI

摘要

近期视频语言模型在视频理解方面展现出巨大潜力，但在事件级感知的精确时序定位上仍存在困难。我们发现视频理解中的两个核心要素（即时序定位与文本响应）存在逻辑层次关系：准确的时序证据定位是生成可靠文本响应的基础。然而现有研究通常以耦合方式处理这两个任务，缺乏清晰的逻辑结构，导致目标函数未能达到最优。对此我们提出因子化学习视角：首先设计D²VLM框架，在解耦双任务学习的同时强调其内在依赖关系。通过采用"先定位后证据引用作答"范式，引入证据标记进行证据定位，该设计超越现有工作对时间戳表示的关注，着重强化事件级视觉语义捕获能力。为促进双任务学习，我们进一步提出新型因子化偏好优化算法（FPO）。与标准偏好优化不同，FPO将概率化时序定位建模显式融入优化目标，实现时序定位与文本响应的联合偏好学习。针对现有数据集缺乏显式时序标注的问题，我们还构建了适用于因子化偏好学习的合成数据集。多任务实验结果表明该方法具有明显优势。项目代码已开源：https://github.com/nusnlp/d2vlm。

English

Recent video-language models have shown great potential for video understanding, but still struggle with accurate temporal grounding for event-level perception. We observe that two main factors in video understanding (i.e., temporal grounding and textual response) form a logical hierarchy: accurate temporal evidence grounding lays the foundation for reliable textual response. However, existing works typically handle these two tasks in a coupled manner without a clear logical structure, leading to sub-optimal objectives. We address this from a factorized learning perspective. We first propose D^2VLM, a framework that decouples the learning of these two tasks while also emphasizing their inherent dependency. We adopt a "grounding then answering with evidence referencing" paradigm and introduce evidence tokens for evidence grounding, which emphasize event-level visual semantic capture beyond the focus on timestamp representation in existing works. To further facilitate the learning of these two tasks, we introduce a novel factorized preference optimization (FPO) algorithm. Unlike standard preference optimization, FPO explicitly incorporates probabilistic temporal grounding modeling into the optimization objective, enabling preference learning for both temporal grounding and textual response. We also construct a synthetic dataset to address the lack of suitable datasets for factorized preference learning with explicit temporal grounding. Experiments on various tasks demonstrate the clear advantage of our approach. Our source code is available at https://github.com/nusnlp/d2vlm.