Light-Omni: 在长期记忆下，智能体视频理解中的反射优于推理

摘要

智能体视频理解赋予模型长期记忆能力，使其能够自主处理并响应连续、长时程的多模态流。然而，高级视频智能体往往依赖"侦探式"迭代推理进行动作控制（如搜索）和证据聚合，导致高昂的成本与延迟。我们认为，这种重度推理本质上是为了弥补全局上下文缺失与检索中的语义对齐不足。本文提出Light-Omni，一种面向反身式轻量化视频理解的多模态智能体框架。该框架通过双上下文状态，在单次前向传递中即时构建所需上下文。首先，我们维护一个全局状态——从情景记忆中持续整合的有限规模多模态脚本来充当Light-Omni的全局上下文。通过层级化合并，它在保留近期细节的同时概括过往事件。其次，基于该全局上下文，我们生成一个参数化潜状态，直接驱动自主动作并产生检索嵌入，且延迟极低。得益于这种耦合设计，Light-Omni在避免迭代推理的同时，实现了语义对齐的检索与反身式响应。大量实验验证了Light-Omni在多个视频基准上的有效性。值得注意的是，相比M3-Agent，它取得了平均2.4%的准确率提升、12.1倍的速度提升以及2.6倍的GPU内存效率提升。此外，它还可作为记忆系统，增强现有大语言模型（MLLMs）的性能与效率。项目页面：https://clare-nie.github.io/Light-Omni。

English

Agentic video understanding equips models with long-term memory to autonomously process and respond to continuous, long-horizon multimodal streams. However, advanced video agents often rely on ``detective-style'' iterative reasoning for action control (e.g., search) and evidence aggregation, incurring prohibitive costs and latency. We argue that such heavy reasoning primarily compensates for the lack of global context and semantic misalignment in retrieval. This paper introduces Light-Omni, a multimodal agent framework for reflexive and lightweight video understanding. It achieves this through dual contextual states that instantly build the required context in a single forward pass. First, we maintain a global state, a finite-sized multimodal script continuously consolidated from episodic memory, serving as the global context for Light-Omni. Through hierarchical merging, it preserves recent details while summarizing past events. Second, conditioned on this global context, we generate a parametric latent state that directly drives autonomous actions and produces retrieval embeddings, with minimal latency. Benefiting from this coupled design, Light-Omni achieves semantically aligned retrieval and reflexive responses while avoiding iterative reasoning. Extensive experiments validate the effectiveness of Light-Omni across multiple video benchmarks. Notably, it outperforms M3-Agent with an average 2.4% accuracy gain, a 12.1times speedup, and a 2.6times improvement in GPU memory efficiency. Furthermore, it serves as a memory system to enhance both the performance and efficiency of existing MLLMs. Project page: https://clare-nie.github.io/Light-Omni.