ShallowStream：串流影片理解之先淺層索引後深層解答

摘要

串流影片理解是實現具身智能、自動駕駛、工業監控、監視預警與可穿戴助理等真實世界應用中的關鍵能力。然而，使用多模態大型語言模型（MLLMs）處理連續影片串流在計算上相當昂貴。現有研究已嘗試透過視覺令牌剪枝、令牌合併、量化、按需幀檢索與上下文卸載等方式來降低串流開銷。然而，多數現有方法忽略了模型深度這一維度。對傳入的幀重複執行全深度 MLLM 預填充，其成本極其高昂，既會產生可觀的計算開銷，也會使 KV 快取以與預填充深度成正比的速度增長。為應對這些挑戰，我們提出 ShallowStream——一個新穎的框架，利用 MLLM 的淺層同時進行幀編碼與檢索索引建構。在串流處理期間，ShallowStream 利用淺層的 KV 快取維護一個常駐的輕量級索引。在查詢階段回答時，我們利用淺層產生的注意力分數對上下文幀進行評分，並採用多樣性感知的選取策略，以檢索精確且全面的證據。ShallowStream 的表現可與現有最強的串流方法並駕齊驅，同時將每幀預填充延遲與 10 秒端到端延遲分別降低最多達 52.1 倍與 11.9 倍。我們的程式碼可在 https://github.com/CURRENTF/ShallowStream 取得。

English

Streaming video understanding is a critical capability for real-world applications, including embodied intelligence, autonomous driving, industrial monitoring, surveillance and early warning, and wearable assistants. However, processing continuous video streams with multimodal large language models (MLLMs) is computationally expensive. Existing efforts have explored reducing streaming overhead through visual token pruning, token merging, quantization, on-demand frame retrieval, and context offloading. However, most existing methods overlook the dimension of model depth. Repeatedly executing full-depth MLLM prefill over incoming frames is prohibitively expensive, incurring substantial computational overhead and causing the KV cache to grow at a rate directly proportional to the prefill depth. To address these challenges, we propose ShallowStream, a novel framework that leverages the shallow layers of an MLLM to simultaneously perform frame encoding and retrieval index building. During stream processing, ShallowStream maintains an always-on lightweight index using the KV cache of shallow layers. During query-time answering, we leverage the attention scores generated by the shallow layers to score context frames and employ a diversity-aware selection strategy to retrieve precise and comprehensive evidence. ShallowStream achieves performance on par with the strongest existing streaming methods, while reducing per-frame prefill latency and 10-second end-to-end latency by up to 52.1x and 11.9x, respectively. Our code is available at https://github.com/CURRENTF/ShallowStream.