EM^2Mem：面向大语言模型的以事件为中心的多模态记忆

摘要

多模态记忆为长视频问答提供了一种可扩展的接口，但现有方法往往以孤立片段的形式检索字幕、帧、转录文本、摘要或图谱事实。这些片段虽然可被检索，却无法直接用于生成：语言模型必须在推理阶段重建跨模态与时间对齐，而该阶段上下文有限且归因困难。为此，我们提出EM^2Mem——一种以事件为中心的多模态记忆框架，在记忆构建过程中将异构证据绑定到事件锚点。每个以事件为索引的记忆单元对齐多模态记录、时间上下文、图谱关联关系、语义事实与来源信息，从而支持围绕锚定的多模态事件进行紧凑的证据读出，而非围绕模态特定的片段。在三个长视频问答基准上，EM^2Mem相较最强记忆基线将平均准确率分别提升2.0、2.4和3.7个百分点，将严格的事件级Top-5证据召回率提升7.0个百分点，并将单查询延迟降低4.67倍、推理词元总量减少63.66%（代码将整合至https://github.com/zjunlp/LightMem）。

English

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).