NeoMME：用於高效微調與推理的單塔式多模態原生多語言基礎編碼器

摘要

多模態模型通常建構於專為生成式視覺語言建模設計的架構之上，一般會將個別預訓練的視覺編碼器與因果語言模型結合。諸如 ColPali 這類視覺文件檢索器，則將這些模型重新用作編碼器，因而把視覺語言模型（VLM）的參數與運算開銷帶入非生成式任務。我們提出 NeoMME，這是一個擁有 260M 與 800M 參數規模的多模態多語言雙向編碼器系列，可在單一雙向 Transformer 編碼器中同時處理多語言文本與原始影像區塊。兩個模型皆以遮蔽式離散擴散文本目標從零開始預訓練，並在多模態樣本中，以可見的影像區塊作為條件。兩者皆支援 16,384 個標記的上下文，足以編碼最多兩張標準 4K UHD 影像。為展現其下游能力，我們以聯合訓練的密集與晚期交互輸出頭對 NeoMME 進行微調。在 ViDoRe v3 基準上，所得到的 NeoMME-Retriever 260M 以 0.523 的 nDCG@10 優於所有參數量嚴格低於 800M 的受評模型，而 NeoMME-Retriever 800M 則達到 0.556。在 NVIDIA L40S 上、輸入影像大小匹配為 2048x2048 時，NeoMME-260M 編碼頁面的吞吐量大約是 ColModernVBERT 的 2 倍。分層標記池化與非對稱量化可將晚期交互多模態文件嵌入壓縮 255 倍，同時保留超過 95% 的基線 nDCG@10。我們將 NeoMME 貢獻至 Hugging Face Transformers，並在 Apache 2.0 許可下發布預訓練骨幹與檢索相容的檢查點，網址為 https://hf.co/collections/Hcompany/neomme。

English

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.