MedSAMix：一种无需训练的医学图像分割模型融合方法

MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation

August 14, 2025

作者: Yanwu Yang, Guinan Su, Jiesi Hu, Francesco Sammarco, Jonas Geiping, Thomas Wolfers

cs.AI

摘要

通用医学图像分割模型因其在多样化任务中展现出的强大泛化能力，已成为一个极具前景的研究方向，在广泛的临床应用场景中显示出巨大潜力。这一潜力的部分推动力来自于通用视觉模型（如Segment Anything Model, SAM）的成功，其激发了多种针对医学分割任务的微调变体的开发。然而，诸如MedSAM等微调变体，受限于相对有限的医学影像数据，这些数据常面临异质性、标注稀缺及分布偏移等挑战，制约了它们在广泛医学分割任务中的泛化能力。鉴于此，我们提出了MedSAMix，一种无需训练的模型融合方法，旨在整合通用模型（如SAM）与专用模型（如MedSAM）的优势，用于医学图像分割。与依赖手动配置且往往导致次优结果的传统模型融合方法不同，我们提出了一种零阶优化方法，以自动发现最优的层级融合方案。此外，针对临床应用需求，我们开发了两种策略，分别通过单任务优化和多目标优化来满足不同场景下对领域特异性和泛化能力的要求。在25项医学分割任务上的广泛评估表明，MedSAMix有效缓解了模型偏差，在领域特异性准确率和泛化能力上均实现了性能提升，分别在专用任务和多任务评估中取得了6.67%和4.37%的改进。

English

Universal medical image segmentation models have emerged as a promising paradigm due to their strong generalizability across diverse tasks, showing great potential for a wide range of clinical applications. This potential has been partly driven by the success of general-purpose vision models such as the Segment Anything Model (SAM), which has inspired the development of various fine-tuned variants for medical segmentation tasks. However, fine-tuned variants like MedSAM are trained on comparatively limited medical imaging data that often suffers from heterogeneity, scarce annotations, and distributional shifts. These challenges limit their ability to generalize across a wide range of medical segmentation tasks. In this regard, we propose MedSAMix, a training-free model merging method that integrates the strengths of both generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical image segmentation. In contrast to traditional model merging approaches that rely on manual configuration and often result in suboptimal outcomes, we propose a zero-order optimization method to automatically discover optimal layer-wise merging solutions. Furthermore, for clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability in different scenarios by single-task optimization and multi-objective optimization respectively. Extensive evaluations on 25 medical segmentation tasks demonstrate that MedSAMix effectively mitigates model bias and consistently improves performance in both domain-specific accuracy and generalization, achieving improvements of 6.67% on specialized tasks and 4.37% on multi-task evaluations.