EM²Mem：面向大型語言模型的事件中心多模態記憶

摘要

多模態記憶為長影片問答提供了可擴展的介面，但現有方法常將字幕、影格、逐字稿、摘要或圖譜事實檢索為孤立片段。雖然這些片段具可搜尋性，卻尚未達到可直接用於生成的程度：語言模型必須在上下文有限且歸因困難的推論階段，重建跨模態與時間性的對齊。我們提出 EM^2Mem，一個以事件為中心的多模態記憶框架，在記憶建構時將異質證據綁定至事件錨點。每個以事件為索引的記憶單元對齊多模態記錄、時間脈絡、圖譜關聯關係、語意事實與來源出處，使得模型能在有依據的多模態事件上進行精簡的證據讀取，而非處理特定模態的片段。在三項長影片問答基準上，EM^2Mem 相較於最強的記憶基線，平均準確度分別提升 2.0、2.4 與 3.7 個百分點；嚴格的事件層級 Top-5 證據召回率提升 7.0 個百分點；每次查詢延遲降低至原來的 1/4.67，總推論 token 數減少 63.66%（程式碼將整合至 https://github.com/zjunlp/LightMem）。

English

Multimodal memory offers a scalable interface for long-video question answering, but existing methods often retrieve captions, frames, transcripts, summaries, or graph facts as isolated fragments. Although searchable, such fragments are not generation-ready: language models must reconstruct cross-modal and temporal alignments at inference time, when context is limited and attribution is difficult. We propose EM^2Mem, an event-centric multimodal memory framework that binds heterogeneous evidence to event anchors during memory construction. Each event-indexed memory cell aligns multimodal records, temporal context, graph-linked relations, semantic facts, and provenance, enabling compact evidence readout over grounded multimodal events rather than modality-specific fragments. Across three long-video QA benchmarks, EM^2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points, improves strict event-level Top-5 evidence recall by 7.0 points, and reduces per-query latency by 4.67 times and total inference tokens by 63.66% (The code will be integrated into https://github.com/zjunlp/LightMem).