NeuPAT：面向語言保留多模態大語言模型的神經元感知可塑性分配調適

摘要

大型語言模型（LLM）的多模態擴展賦予了新的感知能力，但往往會損害預訓練期間獲得的語言智能。在本工作中，我們從內部適應動態的角度研究此現象，並發現預訓練 LLM 中的神經元在多模態學習過程中表現出異質可塑性：有些神經元對保留語言能力至關重要，而另一些則更能適應多模態知識。基於此洞見，我們提出 NeuPAT（神經元感知的可塑性分配調優），這是一個輕量級且與架構無關的框架，在多模態指令調優期間分配神經元級別的更新約束。NeuPAT 使用小規模的探測階段來估計神經元適應模式，並選擇性地保護對語言敏感的神經元，同時通過更具可塑性的神經元促進多模態適應。跨多種 LLM 家族的實驗表明，NeuPAT 在 11 個語言基準測試上恢復了普通調優造成的語言能力退化中的 94.5%，同時保持相當的多模態性能，為保持能力的多模態擴展提供了一種有效方法。

English

Multimodal expansion of large language models (LLMs) enables new perceptual capabilities but often compromises the language intelligence acquired during pretraining. In this work, we investigate this phenomenon from the perspective of internal adaptation dynamics and discover that neurons in pretrained LLMs exhibit heterogeneous plasticity during multimodal learning: some neurons are critical for preserving language capabilities, while others are more adaptive to multimodal knowledge. Based on this insight, we propose NeuPAT (Neuron-aware Plasticity Allocation Tuning), a lightweight and architecture-agnostic framework that allocates neuron-wise update constraints during multimodal instruction tuning. NeuPAT uses a small-scale probing stage to estimate neuron adaptation patterns and selectively protects language-sensitive neurons while promoting multimodal adaptation through more plastic neurons. Experiments across diverse LLM families demonstrate that NeuPAT recovers 94.5\% of the language capability degradation caused by vanilla tuning on 11 language benchmarks while maintaining comparable multimodal performance, providing an effective approach for capability-preserving multimodal expansion.