SALT：基於奇異值適應的低秩變換
SALT: Singular Value Adaptation with Low-Rank Transformation
March 20, 2025
作者: Abdelrahman Elsayed, Sarim Hashmi, Mohammed Elseiagy, Hu Wang, Mohammad Yaqub, Ibrahim Almakky
cs.AI
摘要
醫學影像分割的複雜性要求模型能夠專門捕捉細緻的領域特徵。大型基礎模型提供了顯著的靈活性，但微調這些模型的成本仍然是一個重大障礙。參數高效微調（PEFT）方法，如低秩適應（LoRA），通過低秩矩陣高效更新模型權重，但在選擇的秩不足以捕捉領域特定細微差異時，可能會出現欠擬合問題。相反，基於全秩奇異值分解（SVD）的方法通過修改所有奇異值提供全面的更新，但它們通常缺乏靈活性，並且在不同數據集上表現不一。我們提出了SALT（奇異值適應與低秩轉換），該方法選擇性地使用可訓練的縮放和偏移參數來適應最具影響力的奇異值，同時對剩餘子空間進行低秩更新作為補充。這種混合方法結合了LoRA和SVD的優勢，實現了有效的適應，而無需依賴增加模型規模或深度。在5個具有挑戰性的醫學數據集上進行評估，樣本量從少至20到1000不等，SALT僅使用3.9%的可訓練參數，在Dice係數上比最先進的PEFT（LoRA和SVD）高出2%至5%，即使在低資源環境下也展現出強大的適應能力。SALT的代碼可在以下網址獲取：https://github.com/BioMedIA-MBZUAI/SALT。
English
The complex nature of medical image segmentation calls for models that are
specifically designed to capture detailed, domain-specific features. Large
foundation models offer considerable flexibility, yet the cost of fine-tuning
these models remains a significant barrier. Parameter-Efficient Fine-Tuning
(PEFT) methods, such as Low-Rank Adaptation (LoRA), efficiently update model
weights with low-rank matrices but may suffer from underfitting when the chosen
rank is insufficient to capture domain-specific nuances. Conversely, full-rank
Singular Value Decomposition (SVD) based methods provide comprehensive updates
by modifying all singular values, yet they often lack flexibility and exhibit
variable performance across datasets. We propose SALT (Singular Value
Adaptation with Low-Rank Transformation), a method that selectively adapts the
most influential singular values using trainable scale and shift parameters
while complementing this with a low-rank update for the remaining subspace.
This hybrid approach harnesses the advantages of both LoRA and SVD, enabling
effective adaptation without relying on increasing model size or depth.
Evaluated on 5 challenging medical datasets, ranging from as few as 20 samples
to 1000, SALT outperforms state-of-the-art PEFT (LoRA and SVD) by 2% to 5% in
Dice with only 3.9% trainable parameters, demonstrating robust adaptation even
in low-resource settings. The code for SALT is available at:
https://github.com/BioMedIA-MBZUAI/SALTSummary
AI-Generated Summary