Metis：记忆基础模型

摘要

近年来，AI智能体的进展日益将原生能力内化到其底层基础模型中，催生了多模态基础模型和大型推理模型。然而，智能体记忆仍主要通过外部模块实现，原生记忆能力在很大程度上尚未被探索。本文在这一方向上迈出第一步，提出记忆基础模型，赋予基础模型原生记忆能力。我们从两个视角形式化原生记忆：主干网络内持久且动态演化的记忆状态，以及通过模型计算自主存储和利用信息的原生记忆过程。我们表明，原生内存在架构、端到端优化和效率方面具有优势。基于这一形式化，我们提出Metis，这是记忆基础模型的第一个原型。Metis引入了一种新架构，为基础模型配备原生记忆状态，允许将历史信息压缩到模型中，并通过记忆注意力访问。我们构建了大规模记忆专用训练数据，并引入多种优化目标，通过中期训练习得这些原生记忆过程。Metis的在线记忆维护是无梯度的，记忆更新仅需一次前向传播。在推理时，所有学习到的模型权重保持冻结，而原生记忆状态通过标准前向计算自主转换。通过大量实验，我们表明Metis展现出原生记忆能力，并进一步对其优势、局限和行为进行了详细分析。为促进未来对记忆基础模型的研究，我们发布项目代码和模型检查点。

English

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.