理解与强化任务算术中的权重解耦

Understanding and Enforcing Weight Disentanglement in Task Arithmetic

April 18, 2026

作者: Shangge Liu, Yuehan Yin, Lei Wang, Qi Fan, Yinghuan Shi, Wenbin Li, Yang Gao, Dacheng Tao

cs.AI

摘要

任务算术为编辑预训练模型提供了一种高效且无需重新训练的方法，但其成功缺乏基础性的理论解释。现有“权重解缠”概念描述了任务组合互不干扰的理想结果，但未揭示其根本成因。关键在于：预训练模型（θ₀）或任务向量（τₜ）的何种内在特性促成了这种解缠机制，目前尚待深入探索。本文提出任务特征专化（TFS）——即模型为不同任务分配差异化内部特征的能力——作为核心原理。我们首先证明TFS是权重解缠的充分条件；更重要的是，发现TFS会引发可观测的几何结果：权重向量正交性。这确立了TFS作为功能性目标（解缠）与可度量几何特性（正交性）的共同成因。该关系为我们的方法提供了关键思路：由于抽象的TFS属性难以直接约束，我们可以通过塑造其具体的几何表征（正交性）来促进权重解缠。因此，我们提出OrthoReg——一种简单有效的正则化方法，在微调过程中主动对构成τₜ的权重更新量（ΔW）施加内部正交结构，并从理论上证明该方法能促进解缠。大量实验表明，OrthoReg能持续显著提升多种任务算术方法的性能。代码发布于https://github.com/RL-MIND/OrthoReg。

English

Task arithmetic provides an efficient, training-free way to edit pre-trained models, yet lacks a fundamental theoretical explanation for its success. The existing concept of ``weight disentanglement" describes the ideal outcome of non-interfering task composition but does not reveal its underlying cause. Crucially, what intrinsic properties of the pre-trained model (θ_0) or the task vectors (τ_t) enable this disentanglement remains underexplored. In this paper, we introduce Task-Feature Specialization (TFS), a model's ability to allocate distinct internal features to different tasks, as the fundamental principle. We first prove that TFS is a sufficient condition for weight disentanglement. More importantly, we find that TFS also gives rise to an observable geometric consequence: weight vector orthogonality. This positions TFS as the common cause for both the desired functional outcome (disentanglement) and a measurable geometric property (orthogonality). This relationship provides the key insight for our method: since the abstract TFS property is intractable to enforce directly, we can instead promote weight disentanglement by shaping its concrete geometric consequence, orthogonality. Therefore, we propose OrthoReg, a simple and effective regularization method that actively enforces an internal orthogonal structure on weight updates (ΔW) that constitute τ_t during fine-tuning. And we theoretically prove that OrthoReg promotes disentanglement. Extensive experiments demonstrate that OrthoReg consistently and significantly enhances the performance of various task arithmetic methods. Code is available at https://github.com/RL-MIND/OrthoReg{https://github.com/RL-MIND/OrthoReg}.