全模態密集影片字幕生成之平行化自迴歸解碼

摘要

密集视频描述旨在生成帶有時間定位的視頻事件描述，有助於事件層級的視頻理解與生成。在此領域中，自回歸視頻大語言模型因其強大的生成能力與跨模態建模能力，已成為主流範式。然而，在逐 token 生成模式下產生密集描述，嚴重限制了推論效率，並隨視頻長度與事件密度增加而阻礙可擴展性。本研究提出一個並行化自回歸框架，不僅提升了生成效率，也強化了時間定位描述的性能。我們的關鍵洞察在於利用跨時間事件間的弱局部依賴性，重構因果依賴圖，從而實現無損的並行生成。具體而言，具有弱跨事件依賴關係的 token 可進行並行解碼，而每個事件內部緊密耦合的 token 則保留序列解碼以維持局部語義連貫性。為實現此洞察，我們引入了兩項關鍵元件以達成無損並行解碼：(1) 一種潛在全局規劃機制，可自動學習事件層級結構，產生編碼全局事件間因果關係的緊湊 token，同時自適應聚合事件層級的視聽語義，引導後續的依賴重構與並行解碼；(2) 一種事件分解的並行解碼機制，有效平衡局部關注與全局事件間感知。在多個基準測試上的實驗結果顯示，我們的方法在全模態事件定位與描述上，無論在效率或性能方面均具有明顯優勢。專案網站：https://github.com/showlab/PadCaptioner。

English

Dense video captioning aims to generate temporally grounded descriptions of video events, benefiting both event-level video understanding and generation. In this domain, autoregressive video large language models have emerged as a prevalent paradigm due to their strong generative and cross-modal modeling capacity. However, generating dense captions under the token-by-token paradigm severely limits inference efficiency and hinders scalability as video length and event density increase. In this work, we propose a parallelized autoregressive framework that not only improves generation efficiency but also enhances temporally grounded captioning performance. Our key insight is to exploit the weak local dependencies across temporally distinct events to restructure the causal dependency graph, thereby enabling lossless parallel generation. Specifically, tokens with weak cross-event dependencies can be decoded in parallel, while tightly coupled tokens within each event retain sequential decoding to preserve local semantic coherence. To realize this insight, we introduce two key components for lossless parallel decoding: (1) a latent global planning mechanism that automatically learns the event-level structure and produces compact tokens encoding global inter-event causality while adaptively aggregating event-level audio-visual semantics, guiding subsequent dependency restructuring and parallel decoding; and (2) an event-factorized parallel decoding mechanism that effectively balances local focus with global inter-event awareness. Experiments on various benchmarks demonstrate the clear advantage of our approach in both efficiency and performance in omni-modal event grounding and captioning. Project website: https://github.com/showlab/PadCaptioner.