Patchification 中的比例定律：一張圖片價值 50,176 個標記甚至更多

Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More

February 6, 2025

作者: Feng Wang, Yaodong Yu, Guoyizhe Wei, Wei Shao, Yuyin Zhou, Alan Yuille, Cihang Xie

cs.AI

摘要

自從引入視覺Transformer（ViT）以來，分塊化一直被視為普通視覺架構的一種事實上的圖像標記方法。通過壓縮圖像的空間尺寸，這種方法可以有效地縮短標記序列並降低ViT等普通架構的計算成本。在這項工作中，我們旨在徹底研究基於分塊化的壓縮編碼範式引起的信息損失以及它如何影響視覺理解。我們進行了廣泛的分塊大小縮放實驗，並興奮地觀察到分塊化中的一個有趣的縮放定律：模型可以持續從減小的分塊大小中受益並獲得改善的預測性能，直到達到最小的1x1分塊大小，即像素標記化。這個結論廣泛適用於不同的視覺任務、各種輸入尺度以及不同的架構，如ViT和最近的Mamba模型。此外，作為一個副產品，我們發現使用更小的分塊，任務特定的解碼器頭對於密集預測變得不那麼關鍵。在實驗中，我們成功地將視覺序列擴展到驚人的50,176個標記長度，並在ImageNet-1k基準測試上以基本尺寸模型實現了競爭力的84.6%測試準確度。我們希望這項研究能為未來構建非壓縮視覺模型的工作提供見解和理論基礎。代碼可在https://github.com/wangf3014/Patch_Scaling找到。

English

Since the introduction of Vision Transformer (ViT), patchification has long been regarded as a de facto image tokenization approach for plain visual architectures. By compressing the spatial size of images, this approach can effectively shorten the token sequence and reduce the computational cost of ViT-like plain architectures. In this work, we aim to thoroughly examine the information loss caused by this patchification-based compressive encoding paradigm and how it affects visual understanding. We conduct extensive patch size scaling experiments and excitedly observe an intriguing scaling law in patchification: the models can consistently benefit from decreased patch sizes and attain improved predictive performance, until it reaches the minimum patch size of 1x1, i.e., pixel tokenization. This conclusion is broadly applicable across different vision tasks, various input scales, and diverse architectures such as ViT and the recent Mamba models. Moreover, as a by-product, we discover that with smaller patches, task-specific decoder heads become less critical for dense prediction. In the experiments, we successfully scale up the visual sequence to an exceptional length of 50,176 tokens, achieving a competitive test accuracy of 84.6% with a base-sized model on the ImageNet-1k benchmark. We hope this study can provide insights and theoretical foundations for future works of building non-compressive vision models. Code is available at https://github.com/wangf3014/Patch_Scaling.