面向全模态密集视频描述的并行化自回归解码

摘要

密集视频字幕旨在为视频事件生成带有时间定位的描述，这有助于提升事件级别的视频理解与生成能力。在该领域，自回归视频大型语言模型因其强大的生成能力和跨模态建模能力，已成为主流范式。然而，在逐词生成的范式下生成密集字幕严重限制了推理效率，并随着视频长度和事件密度的增加阻碍了可扩展性。为此，本文提出了一种并行化自回归框架，不仅提升了生成效率，还增强了带有时间定位的字幕生成性能。我们的核心洞察在于：利用时间上不同事件之间较弱的局部依赖性重构因果依赖图，从而实现无损并行生成。具体而言，跨事件依赖性较弱的令牌可以并行解码，而每个事件内部紧密耦合的令牌则保持顺序解码以保留局部语义连贯性。为实现这一洞察，我们引入了两个关键组件用于无损并行解码：(1) 一种潜在全局规划机制，能够自动学习事件级结构，生成紧凑的令牌以编码全局事件间因果关系，同时自适应地聚合事件级的音视频语义，指导后续的依赖重构与并行解码；(2) 一种事件分解的并行解码机制，有效平衡局部关注与全局事件间感知。在多个基准上的实验表明，我们的方法在全模态事件定位与字幕生成方面在效率和性能上均具有明显优势。项目网站：https://github.com/showlab/PadCaptioner。

English

Dense video captioning aims to generate temporally grounded descriptions of video events, benefiting both event-level video understanding and generation. In this domain, autoregressive video large language models have emerged as a prevalent paradigm due to their strong generative and cross-modal modeling capacity. However, generating dense captions under the token-by-token paradigm severely limits inference efficiency and hinders scalability as video length and event density increase. In this work, we propose a parallelized autoregressive framework that not only improves generation efficiency but also enhances temporally grounded captioning performance. Our key insight is to exploit the weak local dependencies across temporally distinct events to restructure the causal dependency graph, thereby enabling lossless parallel generation. Specifically, tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding to preserve local semantic coherence. To realize this insight, we introduce two key components for lossless parallel decoding: (1) a latent global planning mechanism that automatically learns the event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, guiding subsequent dependency restructuring and parallel decoding; and (2) an event-factorized parallel decoding mechanism that effectively balances local focus with global inter-event awareness. Experiments on various benchmarks demonstrate the clear advantage of our approach in both efficiency and performance in omni-modal event grounding and captioning. Project website: https://github.com/showlab/PadCaptioner.