NeuPAT：面向语言保持型多模态大语言模型的神经元感知可塑性分配调优

摘要

大语言模型（LLMs）的多模态扩展赋予了模型新的感知能力，但往往以牺牲预训练阶段习得的语言智能为代价。在本工作中，我们从内部适应动力学的视角探究这一现象，并发现预训练LLM中的神经元在多模态学习过程中表现出异质的可塑性：部分神经元对保持语言能力至关重要，而其他神经元则更适应多模态知识的获取。基于这一洞察，我们提出NeuPAT（神经元感知的可塑性分配微调），一种轻量级且架构无关的框架，在多模态指令微调过程中分配神经元级别的更新约束。NeuPAT通过一个小规模的探测阶段来估计神经元的适应模式，有选择地保护语言敏感神经元，同时通过更具可塑性的神经元促进多模态适应。跨多种LLM家族的实验表明，NeuPAT在11个语言基准上恢复了由标准微调导致的94.5%的语言能力退化，同时保持了相当的多模态性能，为能力保持的多模态扩展提供了一种有效方法。

English

Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.