LoRA如何记忆？大型语言模型微调中的参数记忆法则

摘要

大型语言模型（LLMs）必须持续学习并更新知识，才能在动态的现实环境中保持有效性。尽管低秩适配（LoRA）被广泛用于此类记忆更新，但现有研究主要依赖定性下游评估，对精确参数记忆的定量容量限制及底层动态机制仍缺乏深入探索。为填补这一空白，我们利用LoRA作为潜在空间中的受控记忆容量探针，系统量化精确参数记忆。我们提出了参数记忆定律，这是一个稳健的幂律关系，将损失降低ΔL与有效参数量及序列长度关联起来。在词元级别，细粒度分析揭示了确定性相变，表明在贪婪解码条件下，预测概率p > 0.5是逐字回忆的充分条件。基于这些发现，我们提出了MemFT，一种阈值引导的优化策略，能够动态地将训练预算重新分配给低于阈值的词元。实验评估表明，MemFT能够提升记忆保真度与效率。代码将在https://github.com/zjunlp/ParametricMemoryLaw公开。

English

Large Language Models (LLMs) must continuously learn and update knowledge to remain effective in dynamic real-world environments. While Low-Rank Adaptation (LoRA) is widely used for such memory updates, existing studies mainly rely on qualitative downstream evaluations, leaving the quantitative capacity limits and underlying dynamics of exact parametric memory largely unexplored. To bridge this gap, we employ LoRA as a controlled memory capacity probe within the latent space to systematically quantify exact parametric memory. We introduce the Parametric Memory Law, a robust power law linking loss reduction Delta L to effective parameters and sequence length. At the token level, fine-grained analysis reveals a deterministic phase transition, demonstrating that a prediction probability of p > 0.5 constitutes a sufficient condition for verbatim recall under greedy decoding. Driven by these insights, we introduce MemFT, a threshold-guided optimization strategy that dynamically redistributes the training budget toward sub-threshold tokens. Empirical evaluations demonstrate that MemFT can enhance memory fidelity and efficiency. Code will be released at https://github.com/zjunlp/ParametricMemoryLaw.