大规模记忆解码器：一种预训练的参数化长期记忆

摘要

仅解码器语言模型将长期记忆与推理耦合在单一参数集中，使得难以独立扩展记忆容量。Memory Decoder引入了一个参数化长期记忆模块，但仅在相对较小的规模上进行了研究。在这项工作中，我们提出了大规模Memory Decoder，将记忆模型扩展到6.9B参数，并在300B词元上进行了预训练。在此数据规模下，索引与搜索的总成本使得标准的Faiss流程不可行。我们通过用于Faiss索引和检索的分布式流程，以及稀疏的、按批次加载的kNN分布，解决了这一瓶颈。在不同模型规模上，我们发现，将更多参数分配给记忆，相比单独扩展基础模型，能带来更好的参数-性能权衡。在17个基准测试上，将6.9B的通用记忆与Pythia-410M配对，将其平均得分从29.86提升到37.34，以总参数少39%的优势超过了Pythia-12B（37.24）。对于从0.6B到14B的Qwen3 Base模型，1.7B的领域记忆在每种规模下都将三个领域的平均得分提高了超过9分。总体而言，我们的结果表明，独立扩展预训练记忆为提升语言模型性能提供了一条更具参数效率的路径。

English

Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.