大規模記憶解碼器：一種預訓練的參數化長期記憶

摘要

僅解碼器語言模型將長期記憶與推理耦合在單一參數集中，使得獨立擴展記憶容量變得困難。Memory Decoder 引入了參數化長期記憶模組，但僅在相對較小的規模下進行研究。在本工作中，我們提出了大規模 Memory Decoder，將記憶模型擴展至 6.9B 參數，並在 300B 詞元上進行預訓練。在此資料規模下，索引與搜尋的總成本使得標準 Faiss 管線不可行。我們透過分散式的 Faiss 索引與檢索管線，以及 kNN 分佈的稀疏批次式載入，來解決此瓶頸。在各種模型規模下，我們發現將更多參數分配給記憶模組，比僅擴展基礎模型能帶來更佳的參數-效能取捨。在 17 個基準測試上，將 6.9B 通用記憶與 Pythia-410M 結合，可將其平均分數從 29.86 提升到 37.34，以少 39% 的總參數超越了 Pythia-12B (37.24)。對於從 0.6B 到 14B 的 Qwen3 Base 模型，1.7B 領域記憶模組在每個規模下都能將三個領域的平均分數提升超過 9 分。總體而言，我們的結果證明，獨立擴展預訓練記憶為提升語言模型效能提供了更具參數效率的途徑。

English

Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.