LoRA如何記憶？大型語言模型微調中的參數記憶法則

摘要

大语言模型（LLMs）必须持续学习并更新知识，才能在动态的真实环境中保持有效性。尽管低秩适应（LoRA）被广泛用于此类记忆更新，但现有研究主要依赖定性下游评估，对精确参数记忆的量化容量限制及其底层动态机制仍缺乏深入探索。为填补这一空白，我们将LoRA作为潜空间中的受控记忆容量探针，系统性地量化精确参数记忆。我们提出了参数记忆定律（Parametric Memory Law），即一个将损失减少ΔL与有效参数及序列长度关联起来的稳健幂律关系。在词元级别上，细粒度分析揭示了确定性的相变现象：在贪心解码下，预测概率p>0.5构成了逐字回忆的充分条件。基于这些发现，我们引入MemFT——一种阈值引导的优化策略，该策略将训练预算动态重新分配给阈值以下的词元。实证评估表明，MemFT能够增强记忆保真度与效率。代码将发布于https://github.com/zjunlp/ParametricMemoryLaw。

English

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.