HyperLLaVA：用于多模态大型语言模型的动态视觉和语言专家调整

HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

March 20, 2024

作者: Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, Yueting Zhuang

cs.AI

摘要

最近的进展表明，扩展多模态大型语言模型（MLLMs）有效地提升了在下游多模态任务上的性能。目前流行的MLLM范式，例如LLaVA，通过使用静态视觉-语言映射器将视觉特征转换为类似文本的标记，从而使静态LLMs能够通过视觉指导调整来理解视觉信息的能力。尽管有希望，但共享相同参数的静态调整策略可能会限制在不同下游多模态任务中的性能。鉴此，我们引入了HyperLLaVA，它涉及投影器和LLM参数的自适应调整，分别结合动态视觉专家和语言专家。这些专家源自HyperNetworks，通过视觉和语言指导生成自适应参数变化，从而实现两阶段训练中的动态投影器和LLM建模。 我们的实验表明，我们的解决方案在现有的MLLM基准测试中明显优于LLaVA，包括MME、MMBench、SEED-Bench和LLaVA-Bench。我们的项目可在以下链接找到：https://github.com/DCDmllm/HyperLLaVA。

English

Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks. The prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into text-like tokens using a static vision-language mapper, thereby enabling static LLMs to develop the capability to comprehend visual information through visual instruction tuning. Although promising, the static tuning strategy~The static tuning refers to the trained model with static parameters. that shares the same parameters may constrain performance across different downstream multimodal tasks. In light of this, we introduce HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters, in conjunction with a dynamic visual expert and language expert, respectively. These experts are derived from HyperNetworks, which generates adaptive parameter shifts through visual and language guidance, enabling dynamic projector and LLM modeling in two-stage training. Our experiments demonstrate that our solution significantly surpasses LLaVA on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench. ~Our project is available on the link https://github.com/DCDmllm/HyperLLaVA.