多模式任務向量實現多樣本多模式上下文學習。

Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning

June 21, 2024

作者: Brandon Huang, Chancharik Mitra, Assaf Arbelle, Leonid Karlinsky, Trevor Darrell, Roei Herzig

cs.AI

摘要

最近交錯式大型多模型（LMMs）在少樣本學習中的成功表明，在具有許多範例的情境學習（ICL）可能對於學習新任務是有前景的。然而，這種多樣本多模式ICL設置存在一個關鍵問題：它在預訓練時設定的模型上下文長度基本上是有限的。這個問題在多模式領域尤為突出，該領域處理文本和圖像，需要額外的標記。這促使了需要一種多模式方法，將許多樣本壓縮成較少的標記而無需微調。在這項工作中，我們通過利用多模式任務向量（MTV）實現了LMMs執行多模式、多樣本情境學習，這些MTV是在模型的注意力頭中壓縮的情境範例的緊湊隱式表示。具體來說，我們首先證明了LMMs中存在這樣的MTV，然後利用這些提取的MTV來實現各種視覺和語言任務的多樣本情境學習。我們的實驗表明，MTV可以隨著壓縮樣本數量的增加而提高性能，並且可以推廣到類似的跨領域任務，而無需進行額外的上下文長度進行推斷。

English

The recent success of interleaved Large Multimodal Models (LMMs) in few-shot learning suggests that in-context learning (ICL) with many examples can be promising for learning new tasks. However, this many-shot multimodal ICL setting has one crucial problem: it is fundamentally limited by the model's context length set at pretraining. The problem is especially prominent in the multimodal domain, which processes both text and images, requiring additional tokens. This motivates the need for a multimodal method to compress many shots into fewer tokens without finetuning. In this work, we enable LMMs to perform multimodal, many-shot in-context learning by leveraging Multimodal Task Vectors (MTV)--compact implicit representations of in-context examples compressed in the model's attention heads. Specifically, we first demonstrate the existence of such MTV in LMMs and then leverage these extracted MTV to enable many-shot in-context learning for various vision-and-language tasks. Our experiments suggest that MTV can scale in performance with the number of compressed shots and generalize to similar out-of-domain tasks without additional context length for inference.