大语言模型持续自适应中的记忆库压缩技术

Memory Bank Compression for Continual Adaptation of Large Language Models

January 2, 2026

作者: Thomas Katraouras, Dimitrios Rafailidis

cs.AI

摘要

大型语言模型（LLM）已成为诸多日常应用的核心技术。然而随着数据动态演进，其知识体系会快速过时。持续学习旨在为LLM注入新信息的同时保留既有知识。尽管全参数微调等方法能够整合新数据，但其计算成本高昂且易引发灾难性遗忘——即旧有知识被覆盖的问题。基于记忆增强的方法通过为LLM配备记忆库（即存储信息以供后续调用的外部记忆模块）来解决这一难题。但这类方法存在关键局限：当现实场景中出现大规模数据流时，记忆库会持续膨胀。本文提出MBC模型，该模型通过在线自适应学习期间的码本优化策略实现记忆库压缩。为确保学习稳定性，我们还引入了在线重置机制以防止码本坍缩。此外，我们在LLM的注意力层采用键值低秩自适应技术，从而高效利用压缩后的记忆表征。基于基准问答数据集的实验表明，与最具竞争力的基线方法相比，MBC可将记忆库体积压缩至0.3%，同时在线自适应学习期间保持高记忆保持准确率。代码已开源于https://github.com/Thomkat/MBC。

English

Large Language Models (LLMs) have become a mainstay for many everyday applications. However, as data evolve their knowledge quickly becomes outdated. Continual learning aims to update LLMs with new information without erasing previously acquired knowledge. Although methods such as full fine-tuning can incorporate new data, they are computationally expensive and prone to catastrophic forgetting, where prior knowledge is overwritten. Memory-augmented approaches address this by equipping LLMs with a memory bank, that is an external memory module which stores information for future use. However, these methods face a critical limitation, in particular, the memory bank constantly grows in the real-world scenario when large-scale data streams arrive. In this paper, we propose MBC, a model that compresses the memory bank through a codebook optimization strategy during online adaptation learning. To ensure stable learning, we also introduce an online resetting mechanism that prevents codebook collapse. In addition, we employ Key-Value Low-Rank Adaptation in the attention layers of the LLM, enabling efficient utilization of the compressed memory representations. Experiments with benchmark question-answering datasets demonstrate that MBC reduces the memory bank size to 0.3% when compared against the most competitive baseline, while maintaining high retention accuracy during online adaptation learning. Our code is publicly available at https://github.com/Thomkat/MBC.