多模态模型差异分析用于特征发现与控制

摘要

多模态大语言模型（MLLMs）展现出强大的视觉理解能力，然而导致这些行为的内在特征仍然难以识别、审计或控制。尽管稀疏自编码器（SAEs）将隐藏状态分解为可解释的特征方向，适用于事后检查，但这类分解既难以直接分离出多模态训练所改变的特征，也难以直接用于针对性控制。我们提出MMDiff，一种多模态模型差分框架，该框架训练多模态SAEs并将其转化为特征级接口，用于发现和控制多模态行为。MMDiff支持三种用途：（i）特征分离，通过将基础语言模型SAE与其多模态适配版本进行差分，识别多模态训练所改变的特征；（ii）任务特定特征检测，通过逐词元对比激活分析来分离因果特征；（iii）特征级控制，通过因果性地移除或操控所发现的特征方向。我们为三个MLLM系列模型族——LLaVA-MORE、PaliGemma 2和InternVL3.5——训练多模态SAEs，并在视觉空间理解、多模态安全性和OCR任务上进行评估。MMDiff发现稀疏且因果特定的特征，移除这些特征可选择性地使目标行为性能平均下降12%（空间任务）和17%（OCR），并使多模态安全攻击的攻击成功率降低24%，同时对VQA性能无影响。操控这些特征相比标准的单层引导基线，在空间和OCR准确率上分别平均提升+3.6%和+1.8%。这些结果表明，多模态SAEs不仅可以作为可解释性工具，还可以作为审计、引导和控制MLLM行为的机制，从而推动更安全、更强能力的生成。

English

Multimodal Large Language Models (MLLMs) exhibit strong visual understanding, yet the internal features that cause these behaviors remain difficult to identify, audit, or control. While applicable to post-hoc inspection, hidden states that are decomposed into interpretable feature directions using sparse autoencoders (SAEs) neither readily isolate which features are changed by multimodal training, nor are they directly useful for targeted control. We introduce MMDiff, a multimodal model-diffing framework that trains multimodal SAEs and turns them into feature-level interfaces for discovering and controlling multimodal behavior. MMDiff supports three uses: (i) feature isolation, by diffing a base-LM SAE against its multimodal-adapted counterpart to identify features altered by multimodal training; (ii) task-specific feature detection, via per-token contrastive firing analysis that isolates causal features; and (iii) feature-level control, by causally removing or steering the discovered feature directions. We train multimodal SAEs for three MLLM families, LLaVA-MORE, PaliGemma 2, and InternVL3.5, and evaluate on visual-spatial understanding, multimodal safety, and OCR. MMDiff discovers sparse, causally specific features whose removal selectively degrades target behaviors by an average of 12% on spatial tasks and 17% on OCR, and reduces attack success rate by 24% on multimodal safety attacks, with no impact on VQA performance. Steering these features improves spatial and OCR accuracy by +3.6% and +1.8% on average over a standard single-layer steering baseline. These results show that multimodal SAEs can serve not only as interpretability tools, but as mechanisms for auditing, steering, and controlling MLLMs behavior toward safer and more capable generations.