PaLI-3 视觉语言模型：更小、更快、更强

摘要

本文介绍了PaLI-3，这是一个更小、更快、更强大的视觉语言模型（VLM），与大小为其10倍的类似模型相比表现出色。为了实现这一强大性能，我们比较了使用分类目标预训练的视觉Transformer（ViT）模型和对比性预训练模型（SigLIP）。我们发现，虽然在标准图像分类基准测试上表现略有下降，但基于SigLIP的PaLI在各种多模态基准测试中表现优越，尤其是在定位和视觉文本理解方面。我们将SigLIP图像编码器扩展到20亿参数，并在多语言跨模态检索上取得了新的最先进水平。我们希望，仅有50亿参数的PaLI-3能重新点燃对复杂VLM基础组成部分的研究，并推动新一代规模化模型的发展。

English

This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this strong performance, we compare Vision Transformer (ViT) models pretrained using classification objectives to contrastively (SigLIP) pretrained ones. We find that, while slightly underperforming on standard image classification benchmarks, SigLIP-based PaLI shows superior performance across various multimodal benchmarks, especially on localization and visually-situated text understanding. We scale the SigLIP image encoder up to 2 billion parameters, and achieves a new state-of-the-art on multilingual cross-modal retrieval. We hope that PaLI-3, at only 5B parameters, rekindles research on fundamental pieces of complex VLMs, and could fuel a new generation of scaled-up models.

PaLI-3 视觉语言模型：更小、更快、更强

PaLI-3 Vision Language Models: Smaller, Faster, Stronger

摘要

Support