MoE-ViE：用於高效圖像與視頻理解的混合專家視覺編碼器

摘要

視覺編碼器是視覺-語言模型的關鍵組成部分，有效擴展其容量可顯著提升性能。然而，密集擴展會增加計算成本與推論延遲。混合專家（Mixture-of-Experts, MoE）架構提供了一個極具吸引力的替代方案，已在大型語言模型中實現高效擴展，但在最先進（State-of-the-Art, SOTA）水準下，CLIP風格的視覺編碼器之MoE設計空間仍未得到充分探索。在本研究中，我們系統性地探討了MoE設計在視覺編碼器擴展中的應用，發現細粒度MoE拓撲結構相較於密集對應模型與標準MoE模型均能帶來顯著增益。我們進一步提出了一種無輔助損失的平衡變體，以提升專家利用率，並設計了專門的MoE內核以減輕推論延遲開銷。為在保留影像知識的同時增強影片能力，我們引入了幀級蒸餾搭配新穎的凍結機制。我們預訓練了一系列不同規模的混合專家視覺編碼器（Mixture-of-Experts Vision Encoders, MoE-ViE），所有模型均持續優於其密集對應模型。我們最大的模型以76%的延遲達到了規模為其1.7倍之最先進編碼器的零樣本性能。在與大型語言模型對齊後，MoE-ViE在影像與影片基準測試上超越了所有對比編碼器，包括激活參數量高達5倍的模型。程式碼已公開於 https://github.com/facebookresearch/moe_vie。

English

Vision encoders are a critical component of vision-language models, and scaling their capacity effectively improves performance. However, dense scaling increases compute cost and inference latency. Mixture-of-Experts (MoE) architectures offer a compelling alternative, having enabled efficient scaling in LLMs, yet the MoE design space for CLIP-style vision encoders remains underexplored at State-of-the-Art (SOTA) levels. In this work, we systematically study MoE designs for vision encoder scaling and find that fine-grained MoE topologies yield substantial gains over both dense and standard MoE counterparts. We further propose an auxiliary-loss-free balancing variant for better expert utilization, and design a specialized MoE kernel to mitigate inference latency overhead. To enhance video capabilities while preserving image knowledge, we introduce frame-level distillation paired with a novel freezing mechanism. We pretrain a series of Mixture-of-Experts Vision Encoders (MoE-ViE) across a range of sizes, all consistently outperforming their dense counterparts. Our largest model matches the zero-shot performance of a SOTA encoder 1.7x its size at 76% of its latency. When aligned with an LLM, MoE-ViE surpasses all compared encoders on image and video benchmarks, including those with up to 5x more activated parameters. Code is available at https://github.com/facebookresearch/moe_vie.