TULIP：邁向統一的語言-圖像預訓練

摘要

儘管近期圖像-文本對比模型如CLIP和SigLIP取得了成功，這些模型在處理需要高保真圖像理解的視覺中心任務時，如計數、深度估計和細粒度物體識別，往往表現欠佳。這些模型通過執行語言對齊，傾向於優先考慮高層次語義而非視覺理解，從而削弱了其圖像理解能力。另一方面，專注於視覺的模型擅長處理視覺信息，但在理解語言方面存在困難，這限制了它們在語言驅動任務中的靈活性。在本研究中，我們引入了TULIP，一個開源的、可直接替換現有CLIP類模型的方案。我們的方法利用生成式數據增強、增強的圖像-圖像和文本-文本對比學習，以及圖像/文本重建正則化，來學習細粒度的視覺特徵，同時保持全局語義對齊。我們的方案，參數量超過10億，在多個基準測試中超越了現有的最先進（SOTA）模型，在ImageNet-1K上建立了新的SOTA零樣本性能，在RxRx1的少樣本分類線性探測中相比SigLIP提升了最多2倍，並改進了視覺-語言模型，在MMVP上得分超過SigLIP的3倍。我們的代碼/檢查點可在https://tulip-berkeley.github.io獲取。

English

Despite the recent success of image-text contrastive models like CLIP and SigLIP, these models often struggle with vision-centric tasks that demand high-fidelity image understanding, such as counting, depth estimation, and fine-grained object recognition. These models, by performing language alignment, tend to prioritize high-level semantics over visual understanding, weakening their image understanding. On the other hand, vision-focused models are great at processing visual information but struggle to understand language, limiting their flexibility for language-driven tasks. In this work, we introduce TULIP, an open-source, drop-in replacement for existing CLIP-like models. Our method leverages generative data augmentation, enhanced image-image and text-text contrastive learning, and image/text reconstruction regularization to learn fine-grained visual features while preserving global semantic alignment. Our approach, scaling to over 1B parameters, outperforms existing state-of-the-art (SOTA) models across multiple benchmarks, establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot classification, and improving vision-language models, achieving over 3times higher scores than SigLIP on MMVP. Our code/checkpoints are available at https://tulip-berkeley.github.io

TULIP：邁向統一的語言-圖像預訓練

TULIP: Towards Unified Language-Image Pretraining

摘要

Summary

Support

Support