UltraViT：用於大型視覺語言模型的延遲最佳化裝置端視覺編碼器

摘要

大型视觉語言模型（LVLMs）仍受龐大計算開銷所限，阻礙其部署於資源受限的邊緣設備。儘管壓縮LVLMs的努力主要集中於視覺標記減少或更小的語言模型，視覺編碼器卻在很大程度上被忽略，通常以單體式、計算密集的特徵提取器部署。此外，此前並無任何研究設計出直接針對設備上延遲進行優化的LVLMs視覺編碼器。本文提出UltraViT，這是一款專為LVLMs設計並針對設備上性能進行優化的視覺編碼器。具體而言，通過考慮實際設備上的延遲，我們系統性地設計了一種金字塔式架構，該架構在宏塊層級策略性地整合並適配異質空間混合器。此外，為預訓練UltraViT，我們提出了一種新穎的兩階段生成式預訓練策略：首先通過密集蒸餾培養豐富的空間特徵，隨後由混合容量的凍結大型語言模型提供直接生成式監督。與標準的對比學習及自監督學習相比，我們證明該預訓練方法對於實現UltraViT所需的、用於後續LVLM生成式多模態對齊的高階語義基礎更為有效。大量實驗表明，我們以設備上延遲為導向的設計結合量身訂製的訓練策略，為高效LVLM編碼建立了新的最佳基準，在運行於設備上且速度提升近1.7倍的同時，顯著優於現有的以編碼器為中心的基線方法。

English

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.