MoE-ViE：基于混合专家视觉编码器的高效图像与视频理解

摘要

视觉编码器是视觉-语言模型的关键组成部分，扩展其容量能有效提升性能。然而，密集扩展会增加计算成本和推理延迟。混合专家（Mixture-of-Experts, MoE）架构提供了一种颇具吸引力的替代方案，已在大型语言模型中实现了高效扩展，但在最先进（State-of-the-Art, SOTA）水平下，针对CLIP风格视觉编码器的MoE设计空间仍未得到充分探索。在本工作中，我们系统地研究了用于视觉编码器扩展的MoE设计，发现细粒度MoE拓扑相比密集对应模型和标准MoE模型都能带来显著提升。我们进一步提出一种无辅助损失的均衡变体，以改善专家利用率，并设计了一个专门的MoE内核以减轻推理延迟开销。为在保留图像知识的同时增强视频能力，我们引入了帧级蒸馏，并结合一种新颖的冻结机制。我们预训练了一系列不同规模的混合专家视觉编码器（Mixture-of-Experts Vision Encoders, MoE-ViE），所有模型均一致优于其密集对应版本。我们最大的模型在零样本性能上匹配了规模为其1.7倍的最先进编码器，同时延迟仅为后者的76%。当与大型语言模型对齐时，MoE-ViE在图像和视频基准测试上超越了所有对比编码器，包括那些激活参数多至5倍的编码器。代码可在 https://github.com/facebookresearch/moe_vie 获取。

English

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.