ORBIT：透過源頭調控合併保留生成式檢索中的基礎語言能力

ORBIT: Preserving Foundational Language Capabilities in GenRetrieval via Origin-Regulated Merging

May 12, 2026

作者: Neha Verma, Nikhil Mehta, Shao-Chuan Wang, Naijing Zhang, Alicia Tsai, Li Wei, Lukasz Heldt, Lichan Hong, Ed Chi, Xinyang Yi

cs.AI

摘要

儘管大型語言模型（LLM）的開發進展迅速，但針對特定任務進行微調時，往往會導致其通用的語言推理能力出現災難性遺忘。本研究探討並解決生成式檢索（GenRetrieval）任務中的這一挑戰。在生成式檢索微調過程中，我們發現這種遺忘迅速發生，且與微調後模型與原始模型參數之間的距離相關。基於這些觀察，我們提出ORBIT，一種新穎的方法，透過主動追蹤微調後權重與初始模型權重之間的距離，並在生成式檢索微調過程中，當此模型間距離超過最大閾值時，採用權重平均策略來限制模型漂移。結果顯示，ORBIT能保留大量的文本與檢索效能，優於常見的持續學習基線以及同樣採用權重平均的相關正則化方法。

English

Despite the rapid advancements in large language model (LLM) development, fine-tuning them for specific tasks often results in the catastrophic forgetting of their general, language-based reasoning abilities. This work investigates and addresses this challenge in the context of the Generative Retrieval (GenRetrieval) task. During GenRetrieval fine-tuning, we find this forgetting occurs rapidly and correlates with the distance between the fine-tuned and original model parameters. Given these observations, we propose ORBIT, a novel approach that actively tracks the distance between fine-tuned and initial model weights, and uses a weight averaging strategy to constrain model drift during GenRetrieval fine-tuning when this inter-model distance exceeds a maximum threshold. Our results show that ORBIT retains substantial text and retrieval performance by outperforming both common continual learning baselines and related regularization methods that also employ weight averaging.