B空间拥塞：面向LoRA模型融合的共享方向校准

摘要

合併獨立訓練的LoRA適配器是聯合多任務訓練的實用替代方案，但常導致性能下降。現有方法通常將LoRA更新量ΔW=BA視為單一對象，未區分兩個LoRA矩陣。我們發現合併干擾的主要來源是輸出端矩陣B：跨任務時B會反覆使用少量共享方向，而A則保持較高的任務特定性。這導致合併後的適配器過度強調共享方向，使任務特定信息丟失。我們提出Pico（輸出空間預合併干擾校正法），這種無需數據的方法通過在合併前縮減B矩陣的過度共享方向，再重新調整合併後的更新量來實現校正。Pico可直接嵌入現有合併方法（如任務算術法、TIES和TSV-M）。在數學、編程、金融和醫療領域的八個基準測試中，Pico較基礎方法平均準確率提升3.4-8.3個百分點，達成最佳綜合平均性能。該方法還使合併適配器性能超越使用全量任務數據訓練的LoRA。這些結果表明，分開處理兩個LoRA矩陣能有效提升合併效果。

English

Merging separately trained LoRA adapters is a practical alternative to joint multi-task training, but it often hurts performance. Existing methods usually treat the LoRA update ΔW = BA as a single object and do not distinguish the two LoRA matrices. We show that the main source of LoRA merge interference comes from the output-side matrix B. Across tasks, B repeatedly uses a small set of shared directions, while A remains much more task-specific. As a result, the merged adapter overemphasizes these shared directions, and task-specific information is lost. We propose Pico (Pre-merge interference calibration in output-space), a data-free method that calibrates B before merge by downscaling over-shared directions and then rescaling the merged update. Pico plugs directly into existing merging methods such as Task Arithmetic, TIES, and TSV-M. Across eight different benchmarks from math, coding, finance, and medical domains, Pico improves average accuracy by 3.4-8.3 points over the corresponding base method and achieves the best overall average performance. Pico also enables merged adapters to outperform the LoRA trained with all task data. These results show that LoRA merging works better when the two LoRA matrices are treated separately.

B空间拥塞：面向LoRA模型融合的共享方向校准

Crowded in B-Space: Calibrating Shared Directions for LoRA Merging

摘要

Support