HyperLLaVA：用於多模態大型語言模型的動態視覺和語言專家調整

HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

March 20, 2024

作者: Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, Yueting Zhuang

cs.AI

摘要

最近的進展表明，擴展多模式大型語言模型（MLLMs）有效地提升了在下游多模式任務上的表現。主流的MLLM範式，例如LLaVA，通過靜態視覺-語言映射器將視覺特徵轉換為類似文本的標記，從而使靜態LLMs能夠通過視覺指導調整來理解視覺信息。儘管有潛力，但共享相同參數的靜態調整策略可能會限制在不同下游多模式任務中的表現。鑑於此，我們引入了HyperLLaVA，其中包括對投影機和LLM參數進行自適應調整，分別搭配動態視覺專家和語言專家。這些專家來自於HyperNetworks，通過視覺和語言指導生成自適應參數變化，實現了在兩階段訓練中的動態投影機和LLM建模。 我們的實驗表明，我們的解決方案在現有的MLLM基準上明顯優於LLaVA，包括MME、MMBench、SEED-Bench和LLaVA-Bench。我們的項目可在以下鏈接找到：https://github.com/DCDmllm/HyperLLaVA。

English

Recent advancements indicate that scaling up Multimodal Large Language Models (MLLMs) effectively enhances performance on downstream multimodal tasks. The prevailing MLLM paradigm, e.g., LLaVA, transforms visual features into text-like tokens using a static vision-language mapper, thereby enabling static LLMs to develop the capability to comprehend visual information through visual instruction tuning. Although promising, the static tuning strategy~The static tuning refers to the trained model with static parameters. that shares the same parameters may constrain performance across different downstream multimodal tasks. In light of this, we introduce HyperLLaVA, which involves adaptive tuning of the projector and LLM parameters, in conjunction with a dynamic visual expert and language expert, respectively. These experts are derived from HyperNetworks, which generates adaptive parameter shifts through visual and language guidance, enabling dynamic projector and LLM modeling in two-stage training. Our experiments demonstrate that our solution significantly surpasses LLaVA on existing MLLM benchmarks, including MME, MMBench, SEED-Bench, and LLaVA-Bench. ~Our project is available on the link https://github.com/DCDmllm/HyperLLaVA.