Metis：記憶基礎模型

摘要

近年來，AI 智能體的進展日益將原生能力內化至其底層基礎模型中，催生了多模態基礎模型與大型推理模型。然而，智能體記憶仍主要透過外部模組實現，原生記憶能力在很大程度上尚未被探索。在本文中，我們朝此方向邁出第一步，引入記憶基礎模型，賦予基礎模型原生記憶能力。我們從兩個角度形式化原生記憶：骨幹內部的持久且動態演化的記憶狀態，以及透過模型計算自主儲存與利用資訊的原生記憶程序。我們證明，原生記憶在架構、端到端優化與效率方面具有優勢。基於此形式化，我們提出 Metis，即記憶基礎模型的第一個原型。Metis 引入了一種新架構，為基礎模型配備原生記憶狀態，使歷史資訊能被壓縮至模型中，並透過記憶注意力進行存取。我們建構了大規模的記憶專用訓練資料，並引入多種優化目標，以透過中期訓練習得這些原生記憶程序。Metis 的線上記憶維護是無梯度的，記憶更新僅需一次前向傳播。在推論時，所有已學習的模型權重保持凍結，而原生記憶狀態則透過標準前向計算自主轉換。透過大量實驗，我們證明 Metis 展現出原生記憶能力，並進一步詳細分析其優勢、限制與行為。為促進未來對記憶基礎模型的研究，我們釋出我們的專案與模型檢查點。

English

Recent advances in AI agents have increasingly internalized native capabilities into their underlying foundation models, giving rise to multimodal foundation models and large reasoning models. However, agent memory is still primarily implemented through external modules, leaving the native memory capability largely unexplored. In this paper, we take a first step toward this direction by introducing memory foundation models, which empower foundation models with native memory capabilities. We formalize native memory from two perspectives: a persistent and dynamically evolving memory state within the backbone, and native memory procedures that autonomously store and utilize information through model computation. We show that native memory offers advantages in architecture, end-to-end optimization, and efficiency. Based on this formulation, we propose Metis, the first prototype of memory foundation models. Metis introduces a new architecture that equips a foundation model with a native memory state, allowing historical information to be compressed into the model and accessed through memory attention. We construct large-scale memory-specific training data and introduce multiple optimization objectives to acquire these native memory procedures through mid-training. The online memory maintenance of Metis is gradient-free, and the memory update requires only a forward pass. At inference time, all learned model weights remain frozen, while the native memory states are autonomously transformed through standard forward computation. Through extensive experiments, we show that Metis exhibits native memory capabilities and further provide a detailed analysis of its strengths, limitations, and behaviors. To facilitate future research on memory foundation models, we release our project and model checkpoints.