Light-Omni：在具長期記憶的自主式影片理解中，反射優於推理

摘要

智能體視頻理解賦能模型長期記憶，使其能自主處理並回應連續、長時域的多模態串流。然而，先進的視頻智能體通常依賴「偵探式」迭代推理來進行動作控制（例如搜索）與證據聚合，導致高昂的成本與延遲。我們認為，此類繁重推理主要用於補償全域脈絡的缺失以及檢索語義對齊不足。本文提出Light-Omni，一個用於反射式與輕量級視頻理解的多模態智能體框架。它通過雙重脈絡狀態在單次前向傳播中即時構建所需脈絡。首先，我們維護一個全域狀態，即從情景記憶中持續壓縮而成的有限大小多模態腳本，作為Light-Omni的全域脈絡。通過層次化合併，它在保留近期細節的同時總結過往事件。其次，基於此全域脈絡，我們生成一個參數化潛狀態，直接驅動自主動作並產出檢索嵌入，延遲極低。得益於這種耦合設計，Light-Omni實現了語義對齊檢索與反射式回應，避免了迭代推理。大量實驗驗證了Light-Omni在多個視頻基準上的有效性。值得注意的是，它相較於M3-Agent平均準確率提升2.4%，速度提升12.1倍，GPU記憶體效率提升2.6倍。此外，它可作為記憶系統增強現有MLLM的性能與效率。專案頁面：https://clare-nie.github.io/Light-Omni。

English

Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style'' iterative reasoning for action control (e.g., search) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, a multimodal agent framework for reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain a global state, a finite-sized multimodal script continuously consolidated from episodic memory, serving as the global context for Light-Omni. Through hierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate a parametric latent state that directly drives autonomous actions and produces retrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoiding iterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiple video benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1times speedup, and a 2.6times improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existing MLLMs. Project page: https://clare-nie.github.io/Light-Omni.