喚醒觸覺！MLLMs中的遮罩隔離觸覺對齊學習

摘要

觸覺提供了感知材料固有屬性（如摩擦力和柔順度）所需的物理基礎，而這些屬性僅靠視覺往往無法分辨。然而，近期為多模態大語言模型配備此觸覺感知能力的努力，卻暴露出一個零和取捨：緊湊模型的有限參數預算迫使在獲取新感知模態與維持既有視覺語言推理能力之間做出選擇。我們提出 Splash——一種基於掩碼隔離的觸覺對齊學習框架，適用於多模態大語言模型。Splash 量化了每個預訓練參數的重要性，並將參數空間劃分為休眠子空間與關鍵子空間。在關鍵子空間凍結作為穩定錨點以保護通用視覺知識的同時，Splash 更新隔離的休眠子空間，將觸覺對齊內化至大語言模型中。這種選擇性、無破壞性的擴展有效防止了災難性遺忘，並確保了模態擴展的非破壞性。大量實驗表明，Splash 能在不增加大語言模型部分額外推理開銷的情況下有效實現觸覺推理，在視覺-觸覺基準測試（包括 SSVTP、TVL 和 TacQuad）上展現出最先進性能，同時保持其原有的通用能力。

English

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.