面向大语言模型持续自适应性的记忆库压缩技术

Memory Bank Compression for Continual Adaptation of Large Language Models

January 2, 2026

作者: Thomas Katraouras, Dimitrios Rafailidis

cs.AI

摘要

大型语言模型（LLMs）已成为众多日常应用的核心技术。然而随着数据动态演变，其知识体系会快速过时。持续学习旨在为LLMs注入新信息的同时保留既有知识。尽管全参数微调等方法能够整合新数据，但存在计算成本高、易引发灾难性遗忘（即旧知识被覆盖）的问题。基于记忆增强的方法通过为LLMs配备记忆库——即存储信息以供未来调用的外部记忆模块——来解决这一难题。但这类方法面临关键局限：当现实场景中出现大规模数据流时，记忆库会持续膨胀。本文提出MBC模型，该模型通过在线自适应学习过程中的码本优化策略实现记忆库压缩。为确保学习稳定性，我们还引入了在线重置机制以防止码本坍塌。此外，我们在LLM的注意力层采用键值低秩自适应技术，从而高效利用压缩后的记忆表征。基于基准问答数据集的实验表明，与最具竞争力的基线方法相比，MBC可将记忆库体积压缩至0.3%，同时在线自适应学习过程中保持高记忆保持准确率。代码已开源：https://github.com/Thomkat/MBC。

English

Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question-answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC.