MedSAMix：一種免訓練的模型融合方法應用於醫學影像分割

MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation

August 14, 2025

作者: Yanwu Yang, Guinan Su, Jiesi Hu, Francesco Sammarco, Jonas Geiping, Thomas Wolfers

cs.AI

摘要

通用醫學影像分割模型因其在多樣化任務中的強大泛化能力而嶄露頭角，展現出廣泛臨床應用的巨大潛力。這一潛力部分得益於通用視覺模型（如Segment Anything Model, SAM）的成功，其激發了多種針對醫學分割任務的微調變體的開發。然而，像MedSAM這樣的微調變體在相對有限的醫學影像數據上進行訓練，這些數據往往存在異質性、註釋稀缺以及分佈偏移等問題。這些挑戰限制了它們在廣泛醫學分割任務中的泛化能力。有鑑於此，我們提出了MedSAMix，這是一種無需訓練的模型融合方法，它整合了通用模型（如SAM）和專用模型（如MedSAM）的優勢，用於醫學影像分割。與依賴手動配置且往往導致次優結果的傳統模型融合方法不同，我們提出了一種零階優化方法，以自動發現層級最優的融合方案。此外，針對臨床應用，我們開發了兩種策略，分別通過單任務優化和多目標優化來滿足不同場景下對領域特定性和泛化性的需求。在25個醫學分割任務上的廣泛評估表明，MedSAMix有效減輕了模型偏差，並在領域特定準確性和泛化能力上持續提升性能，在專用任務上實現了6.67%的提升，在多任務評估中提升了4.37%。

English

Universal medical image segmentation models have emerged as a promising paradigm due to their strong generalizability across diverse tasks, showing great potential for a wide range of clinical applications. This potential has been partly driven by the success of general-purpose vision models such as the Segment Anything Model (SAM), which has inspired the development of various fine-tuned variants for medical segmentation tasks. However, fine-tuned variants like MedSAM are trained on comparatively limited medical imaging data that often suffers from heterogeneity, scarce annotations, and distributional shifts. These challenges limit their ability to generalize across a wide range of medical segmentation tasks. In this regard, we propose MedSAMix, a training-free model merging method that integrates the strengths of both generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical image segmentation. In contrast to traditional model merging approaches that rely on manual configuration and often result in suboptimal outcomes, we propose a zero-order optimization method to automatically discover optimal layer-wise merging solutions. Furthermore, for clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability in different scenarios by single-task optimization and multi-objective optimization respectively. Extensive evaluations on 25 medical segmentation tasks demonstrate that MedSAMix effectively mitigates model bias and consistently improves performance in both domain-specific accuracy and generalization, achieving improvements of 6.67% on specialized tasks and 4.37% on multi-task evaluations.