用於長影片理解的線性擴展影片VLM

摘要

视频视觉语言模型（VLMs）正被越来越多地应用于长期和流式场景，然而绝大多数视频编码器仍依赖时空自注意力机制，导致其计算量和延迟随帧数呈二次增长。现有的效率方法虽提升了可扩展性，但相较于完整自注意力机制往往损失精度，例如通过激进的帧/令牌丢弃或粗略的注意力近似。我们提出StateKV——一种推理阶段方法，通过将跨帧上下文承载于固定容量、基于重要性的循环状态中，并辅以用于解码的第二个全量逐帧缓存，从而将预训练的长视频VLM适配为线性时间视频预填充。在三个长视频基准测试及涵盖三个模型家族、多种规模的七个模型上，StateKV的精度接近完整自注意力机制，且持续优于主流的滑动窗口/基于近期信息的流式近似方法，且无需微调或架构修改。StateKV还降低了以FLOPs衡量的视频预填充成本，使得在固定计算预算下能够运行更大模型来获得更强的精度。这些结果表明，这是迈向可扩展长视频理解的一个实践性步骤。

English

Video vision-language models (VLMs) are increasingly used in long-horizon and streaming settings, yet most video encoders still rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. Existing efficiency methods improve scalability but often lose accuracy relative to full self-attention, for example through aggressive frame/token dropping or coarse attention approximations. We introduce StateKV, an inference-time method that adapts pretrained long-video VLMs to linear-time video prefill by carrying cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models spanning three families and multiple scales, StateKV remains close to full self-attention and consistently outperforms dominant sliding-window / recency-based streaming approximations, without fine-tuning or architectural changes. StateKV also reduces video-prefill cost measured FLOPs, enabling stronger accuracy at a fixed compute budget by running larger models. These results suggest a practical step toward scalable long-video understanding.