唤醒触觉！多模态大语言模型中的掩码隔离触觉对齐学习

摘要

触觉提供了感知内在物理属性（如摩擦力和顺应性）所需的物理基础，而这些属性仅凭视觉往往难以分辨。然而，近期为多模态大语言模型（MLLMs）配备这种触觉感知能力的尝试暴露了一个零和权衡：紧凑模型有限的参数预算迫使其在获取新感觉模态与保留既有视觉-语言推理能力之间做出选择。我们提出Splash，一种面向MLLMs的掩码隔离触觉对齐学习框架。Splash量化每个预训练参数的重要性，并将参数空间划分为休眠子空间与关键子空间。在冻结的关键子空间作为稳定锚点以保护通用视觉知识的同时，Splash更新隔离的休眠子空间，将触觉对齐内化至大语言模型。这种选择性的、非破坏性扩展有效防止了灾难性遗忘，并确保了非破坏性的模态扩展。大量实验表明，Splash能够在无需增加大语言模型部分推理开销的情况下有效实现触觉推理，在SSVTP、TVL和TacQuad等视觉-触觉基准测试中展现了最先进的性能，同时保持其原有的通用能力。

English

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.