视觉-语言建模简介

An Introduction to Vision-Language Modeling

May 27, 2024

作者: Florian Bordes, Richard Yuanzhe Pang, Anurag Ajay, Alexander C. Li, Adrien Bardes, Suzanne Petryk, Oscar Mañas, Zhiqiu Lin, Anas Mahmoud, Bargav Jayaraman, Mark Ibrahim, Melissa Hall, Yunyang Xiong, Jonathan Lebensold, Candace Ross, Srihari Jayakumar, Chuan Guo, Diane Bouchacourt, Haider Al-Tahan, Karthik Padthe, Vasu Sharma, Hu Xu, Xiaoqing Ellen Tan, Megan Richards, Samuel Lavoie, Pietro Astolfi, Reyhane Askari Hemmat, Jun Chen, Kushal Tirumala, Rim Assouel, Mazda Moayeri, Arjang Talattof, Kamalika Chaudhuri, Zechun Liu, Xilun Chen, Quentin Garrido, Karen Ullrich, Aishwarya Agrawal, Kate Saenko, Asli Celikyilmaz, Vikas Chandra

cs.AI

摘要

随着大型语言模型（LLMs）近来的流行，人们开始尝试将其扩展到视觉领域。从拥有能够引导我们穿越陌生环境的视觉助手，到使用仅高层文本描述生成图像的生成模型，视觉语言模型（VLM）的应用将显著影响我们与技术的关系。然而，有许多挑战需要解决，以提高这些模型的可靠性。语言是离散的，而视觉则存在于一个更高维度的空间中，其中概念并非总是容易离散化。为了更好地理解将视觉映射到语言背后的机制，我们提出了这篇介绍VLMs的文章，希望能帮助任何希望进入该领域的人。首先，我们介绍了VLMs是什么，它们如何工作以及如何训练它们。然后，我们提出并讨论了评估VLMs的方法。虽然这项工作主要集中在将图像映射到语言上，但我们也讨论了将VLMs扩展到视频的可能性。

English

Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos.