StreamArena：邁向連續性、互動性與長時程的代理式串流影片理解

摘要

在連續、真實世界環境中部署自主多模態代理，需要其攝入無界定的音訊-視覺串流，並維持小時級的記憶。然而，目前的評測主要依賴簡短片段與選擇題形式。此設計使僅處理最後四幀的極簡基線方法得以媲美或超越複雜的串流模型，同時答案選項亦暴露了語言捷徑。我們提出StreamArena，一個針對小時級、互動式串流影片理解所設計的基準測試。StreamArena包含243部完整影片，平均長度為88.8分鐘，以及3,646組經嚴謹註釋的開放式問答對，用以評估即時感知、歷史回顧、主動互動與多模態工具利用等能力。對多種系統的評估揭示了連續互動與長時程多模態理解之間的張力。僅保留近期幀的方法無法回溯遙遠事件；將過往觀察轉化為文字的方法會流失視覺證據；而反覆壓縮視覺記憶的方法則難以隨時間保留細粒度的細節。我們以StreamMind來應對此張力，其為一種兩層式架構，將延遲關鍵的互動與主動監控分配給獨立調度的前端工作者，同時由後端工作者非同步建構持久化的多模態記憶，並執行歷史召回與外部搜尋。StreamMind在所有四項能力上皆優於現有串流基線方法，並透過重用持久化狀態降低查詢至回答的延遲。

English

Deploying autonomous multimodal agents in continuous, real-world environments requires them to ingest unbounded audio-visual streams and maintain hour-scale memory. However, current evaluations predominantly rely on brief clips and multiple-choice formats. This design allows minimal baselines that process only the last four frames to match or surpass complex streaming models, while answer options also expose language shortcuts. We introduce StreamArena, a benchmark for hour-scale, interactive streaming video understanding. StreamArena contains 243 full-length videos averaging 88.8 minutes and 3,646 rigorously annotated, open-ended question-answer pairs that evaluate real-time perception, historical retrospection, proactive interaction, and multimodal tool utilization. Evaluation across diverse systems exposes a tension between continuous interaction and long-horizon multimodal comprehension. Methods that retain only recent frames cannot recover distant events, methods that convert past observations into text lose visual evidence, and methods that repeatedly compress visual memory struggle to preserve fine-grained details over time. We address this tension with StreamMind, a two-tier architecture that assigns latency-critical interaction and proactive monitoring to independently scheduled frontend workers, while backend workers asynchronously construct persistent multimodal memory and perform historical recall and external search. StreamMind outperforms existing streaming baselines across all four capabilities and reduces query-to-answer latency by reusing persistent state.