抖音多模态嵌入模型技术报告

摘要

多模态表示学习是现代人工智能的基石。通过将多模态查询与目标编码为向量，它支撑着工业级搜索与推荐系统，并为现代智能体奠定基础。在抖音、小红书、YouTube等具有复杂模态和海量内容的真实平台中，既需要在十亿级索引下保持高效，也需要具备细粒度判别能力以应对困难匹配。现有的多模态大语言模型（MLLM）嵌入模型难以同时满足这两点。对比模型虽然高效，但依赖的成对监督过于粗糙，难以进行细粒度区分；而基于思维链（CoT）的模型虽通过显式生成提升了判别能力，却因生成开销过大而无法在线服务。我们提出了抖音多模态嵌入模型（Douyin Multimodal Embedding, DME），采用两阶段训练以兼取两者之长。第一阶段进行大规模对比预训练，构建统一的多模态嵌入空间，覆盖广泛的模态与任务。第二阶段通过两种机制补足语义充分性——即嵌入基于检索相关证据，并保留细粒度的对端语义。证据支撑的类型化潜在推理（Evidence-Grounded Typed Latent Reasoning）通过隐空间中的潜在推理来组织检索证据；跨条件重建（Cross-Conditional Reconstruction）则通过跨方向自回归重建来强化对端语义。这两种机制仅在训练阶段生效，且仅引入极小的查询端开销，因此DME的服务效率与标准对比编码器相当。在MMEB-v2基准上，DME的2B与9B变体在同等规模下均达到最先进水平（分别为74.8与78.4），在视频和视觉文档任务上表现尤为突出。在生产环境中，DME在抖音内部离线评测集上取得了2.92%的相对提升，已部署于抖音的生成式搜索、图片搜索和AI搜索等多个场景，并在抖音搜索的在线A/B测试中带来了0.1%的Lifetime（LT）提升。

English

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.