如何教授大型多模态模型新技能

摘要

如何在不抹去已有能力的前提下，教会大型多模态模型（LMMs）新技能？我们研究了在五种目标技能上的顺序微调过程，同时监控了三种模型系列在八个保留基准上的通用能力。我们观察到，在针对特定任务进行窄化微调后，保留任务上表现出的“遗忘”现象在后续阶段会部分恢复。我们将这一行为归因于输出令牌分布的可测量变化，通过一个与遗忘共变的简单计数偏差探针得以显现。基于这一观察，我们提出了两种简单且稳健的调优策略，它们在学习新技能的同时有效限制了性能漂移：（i）仅更新自注意力投影层，以及（ii）仅更新MLP的Gate&Up部分，同时冻结Down投影。在多种模型和任务中，这些选择在显著提升目标技能的同时，基本保持了在保留任务上的性能。相关代码已发布于https://github.com/jessemelpolio/LMM_CL。

English

How can we teach large multimodal models (LMMs) new skills without erasing prior abilities? We study sequential fine-tuning on five target skills while monitoring general ability on eight held-out benchmarks across three model families. We observe that apparent "forgetting" on held-out tasks after narrow fine-tuning can partly recover at later stages. We trace this behavior to a measurable shift in the output token distribution, manifested through a simple counting-bias probe that co-varies with forgetting. Guided by this picture, we identify two simple, robust tuning recipes that learn strongly while limiting drift: (i) updating only the self-attention projection layers, and (ii) updating only the MLP Gate&Up while freezing the Down projection. Across models and tasks, these choices deliver strong target gains while largely preserving held-out performance. Code is available at https://github.com/jessemelpolio/LMM_CL

如何教授大型多模态模型新技能

How to Teach Large Multimodal Models New Skills

摘要

Support