StreamBridge：將您的離線視頻大型語言模型轉變為主動式串流助手

StreamBridge: Turning Your Offline Video Large Language Model into a Proactive Streaming Assistant

May 8, 2025

作者: Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, Ping Huang

cs.AI

摘要

我們提出了StreamBridge，這是一個簡單而有效的框架，能夠無縫地將離線的Video-LLM轉化為支持串流的模型。它解決了將現有模型適應於線上場景時的兩個基本挑戰：(1) 多輪實時理解能力的限制，以及 (2) 缺乏主動回應機制。具體而言，StreamBridge整合了：(1) 一個結合了輪次衰減壓縮策略的記憶緩衝區，支持長上下文的多輪互動，以及 (2) 一個解耦的輕量級激活模型，可以輕鬆整合到現有的Video-LLM中，實現持續的主動回應。為了進一步支持StreamBridge，我們構建了Stream-IT，這是一個專為串流影片理解而設計的大規模數據集，具有交錯的影片-文本序列和多樣的指令格式。大量實驗表明，StreamBridge顯著提升了離線Video-LLM在各種任務中的串流理解能力，甚至超越了GPT-4o和Gemini 1.5 Pro等專有模型。同時，它在標準的影片理解基準測試中也達到了競爭力或更優的表現。

English

We present StreamBridge, a simple yet effective framework that seamlessly transforms offline Video-LLMs into streaming-capable models. It addresses two fundamental challenges in adapting existing models into online scenarios: (1) limited capability for multi-turn real-time understanding, and (2) lack of proactive response mechanisms. Specifically, StreamBridge incorporates (1) a memory buffer combined with a round-decayed compression strategy, supporting long-context multi-turn interactions, and (2) a decoupled, lightweight activation model that can be effortlessly integrated into existing Video-LLMs, enabling continuous proactive responses. To further support StreamBridge, we construct Stream-IT, a large-scale dataset tailored for streaming video understanding, featuring interleaved video-text sequences and diverse instruction formats. Extensive experiments show that StreamBridge significantly improves the streaming understanding capabilities of offline Video-LLMs across various tasks, outperforming even proprietary models such as GPT-4o and Gemini 1.5 Pro. Simultaneously, it achieves competitive or superior performance on standard video understanding benchmarks.