UltraViT：面向大型视觉语言模型的延迟优化端侧视觉编码器

摘要

大型视觉-语言模型（LVLMs）仍受制于庞大的计算开销，难以部署在资源受限的边缘设备上。现有LVLMs压缩工作主要聚焦于视觉令牌压缩或更小的语言模型，但视觉编码器却被严重忽视——其通常以单块计算密集型特征提取器的形式部署。此外，此前尚无研究针对设备端延迟直接优化设计LVLMs的视觉编码器。本文提出UltraViT，一种专为设备端性能显式设计与优化的LVLMs视觉编码器。具体而言，通过考量实际设备端延迟，我们系统性地设计了一种金字塔架构，在宏块层面策略性地集成并适配异质空间混合器。为预训练UltraViT，我们提出新型两阶段生成式预训练策略：首先通过密集蒸馏培育丰富的空间特征，随后由混合容量冻结大语言模型进行直接生成式监督。相较于标准对比学习和自监督学习，我们的预训练方法在实现UltraViT所需的高层级语义锚定方面更为高效，为后续LVLMs训练的生成式多模态对齐奠定基础。大量实验表明，结合设备端延迟感知设计及定制化训练策略，UltraViT在高效LVLMs编码领域达到新高度，显著超越现有以编码器为中心的基线方法，同时以近1.7倍的速度在设备端运行。

English

Large Vision-Language Models (LVLMs) remain bottlenecked by massive computational footprints, precluding their deployment on resource-constrained edge devices. While efforts to compress LVLMs focus heavily on vision token reduction or smaller language models, the vision encoder is largely overlooked, typically deployed as a monolithic, computationally heavy feature extractor. Moreover, there is no previous effort that designs a vision encoder for LVLMs directly optimized for on-device latency. In this paper, we present UltraViT, a vision encoder for LVLMs, explicitly designed and optimized for on-device performance. Specifically, by taking into account real on-device latencies, we systematically design a pyramidal architecture that strategically integrates and adapts heterogeneous spatial mixers at the macro-block level. Furthermore, to pre-train UltraViT, we propose a novel two-stage generative pre-training strategy: cultivating rich spatial features via dense distillation, followed by direct generative supervision from a capacity-mixed frozen LLM. Compared to standard contrastive and SSL, we show that our pre-training is much more effective for achieving high-level semantic grounding for UltraViT needed for the subsequent generative multimodal alignment of LVLM training. Extensive experiments demonstrate that our on-device latency-informed design combined with our tailored training strategy establishes a new state-of-the-art for efficient LVLM encoding, significantly outperforming existing encoder-centric baselines while operating on-device at nearly 1.7xthe speed.