超越检索：面向流式视频理解的渐进式潜在记忆演化

摘要

流式视频理解要求多模态大语言模型（MLLMs）在严格的因果约束和有限记忆条件下处理连续视觉输入并响应用户查询。现有方法通常将历史观测压缩到外部记忆库中，并检索与查询相关的证据作为额外的视觉上下文。尽管这类方法行之有效，但这种“存储-检索”范式始终将历史证据保留为外部视觉上下文，使其无法被内化为一种紧凑且持续演化的潜在记忆，从而难以持续引导流式推理。为弥合这一差距，我们提出LatentStream——一种渐进式潜在工作记忆框架，将流式记忆从“存储-检索”转变为“检索-内化”。具体而言，LatentStream由三个相互协作的组件构成。首先，查询无关的分层流式记忆（Query-agnostic Hierarchical Streaming Memory）在固定记忆预算内，通过Jenks引导的自适应整合，将视觉历史组织为短期、中期和长期三个层次。当用户查询到达时，分层潜在记忆演化（Hierarchical Latent Memory Evolution）为各组潜在记忆标记赋予逐步扩展的记忆感受野，使其能够从各自对应的范围内迭代检索历史证据，并将其内化为长度固定的紧凑潜在记忆。最后，渐进式置信度引导的潜在记忆优化（Progressive Confidence-guided Latent Memory Optimization）根据分组预测熵构建分层递进奖励，并对潜在记忆标记与检索证据进行联合优化，促使流式推理的置信度不断提升。大量实验表明，LatentStream在现有在线与离线视频基准上均取得了新的最佳性能。

English

Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.