NeoMME：一种面向高效微调与推理的单塔原生多模态多语言基础编码器

摘要

多模态模型通常构建于为生成式视觉-语言建模而设计的架构之上，典型做法是将分别预训练的视觉编码器与因果语言模型相结合。视觉文档检索器（如ColPali）将此类模型重新用作编码器，从而在非生成式任务中沿用了视觉语言模型（VLM）的参数与计算开销。我们提出NeoMME，这是一个包含260M与800M参数的多模态、多语言双向编码器系列，可在单一双向Transformer编码器中处理多语言文本和原始图像块。两个模型均以掩码离散扩散文本目标从零开始预训练；对于多模态样本，训练以可见图像块为条件。二者均支持16,384个token的上下文长度，足以编码多达两幅标准4K超高清图像。为展示其下游能力，我们使用联合训练的稠密头部与后期交互头部对NeoMME进行微调。在ViDoRe v3基准上，由此得到的NeoMME-Retriever 260M以0.523的nDCG@10优于所有参数规模严格低于800M的被评估模型，而NeoMME-Retriever 800M达到0.556。在NVIDIA L40S上以统一的2048×2048图像输入尺寸进行测试时，NeoMME-260M的页面编码吞吐量约为ColModernVBERT的2倍。分层token池化与非对称量化可将后期交互多模态文档嵌入压缩255倍，同时保留基准nDCG@10的95%以上。我们将NeoMME贡献至Hugging Face Transformers，并在Apache 2.0许可下于https://hf.co/collections/Hcompany/neomme发布预训练骨干模型及检索兼容检查点。

English

Multimodal models often build on architectures designed for generative vision-language modeling, typically combining separately pretrained vision encoders with causal language models. Visual document retrievers such as ColPali repurpose these models as encoders, carrying over the parameter and compute overhead of a VLM for a non-generative task. We introduce NeoMME, a family of 260M and 800M-parameter Multimodal and Multilingual bidirectional Encoders that process multilingual text and raw image patches in a single bidirectional Transformer encoder. Both models are pretrained from scratch with a masked discrete-diffusion text objective, conditioned on visible image patches for multimodal examples. Both support a 16,384-token context, enough to encode up to two standard 4K UHD images. To demonstrate its downstream capabilities, we fine-tune NeoMME with jointly trained dense and late-interaction heads. On the ViDoRe v3 benchmark, the resulting NeoMME-Retriever 260M outperforms all evaluated models strictly below 800M parameters with 0.523 nDCG@10, while NeoMME-Retriever 800M reaches 0.556. At a matched 2048x2048 image input size on an NVIDIA L40S, NeoMME-260M encodes pages with about 2x the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization compress late-interaction multimodal document embeddings by 255x while preserving over 95% of baseline nDCG@10. We contribute NeoMME to Hugging Face Transformers and release the pretrained backbone and retrieval-compatible checkpoints under Apache 2.0 at https://hf.co/collections/Hcompany/neomme.