抖音多模態嵌入模型技術報告

摘要

多模態表徵學習是現代人工智慧的基石。透過將多模態查詢與目標編碼為向量，它驅動了工業級搜尋與推薦系統，並支撐現代智能體的運作。諸如抖音、小紅書與 YouTube 等真實平台，因其複雜的模態與海量內容，既需要在十億級索引下的高效率，也需要對困難匹配進行細粒度區分。現有的多模態大型語言模型（MLLM）嵌入模型很少能同時滿足這兩項要求。對比式模型雖然高效，但依賴於成對監督，其粒度過粗，難以應對細粒度區分；而基於思維鏈（CoT）的模型雖能透過顯式生成提升區分能力，但在線上服務中卻不切實際。我們提出抖音多模態嵌入（Douyin Multimodal Embedding, DME），這是一個以兩階段訓練來結合兩者優勢的模型。第一階段進行大規模對比式預訓練，建立一個具有廣泛模態與任務覆蓋的統一多模態嵌入空間。第二階段則透過兩種機制補充語義充分性——即嵌入係奠基於檢索相關證據，並保留細粒度的對側語義之性質。其一為證據基底的類型化潛在推理（Evidence-Grounded Typed Latent Reasoning），透過隱空間中的潛在推理來組織檢索證據；其二為跨條件重建（Cross-Conditional Reconstruction），透過跨方向的自迴歸重建來強化對側語義。兩者僅在訓練期間運作，且僅增加微小的查詢端開銷，因此 DME 的服務效率與標準對比式編碼器相同。在 MMEB-v2 上，DME 的 2B 與 9B 變體在可比規模下達到最先進的結果（分別為 74.8 與 78.4），尤其在影片與視覺文件任務上表現突出。在實際部署中，DME 在抖音內部離線評測集上取得 2.92% 的相對提升，已部署於抖音的生成式搜尋、圖像搜尋與 AI 搜尋等場景，並在抖音搜尋的線上 A/B 測試中獲得 0.1% 的 Lifetime（LT）增益。

English

Multimodal representation learning is a cornerstone of modern AI. By encoding multimodal queries and targets into vectors, it powers industrial search and recommendation and underpins modern agents. Real-world platforms with complex modalities and massive-scale content, such as Douyin, Xiaohongshu, and YouTube, demand both efficiency under billion-scale indexing and fine-grained discrimination for hard matching. Existing MLLM embedding models rarely satisfy both. Contrastive models are efficient but rely on pair-level supervision too coarse for fine-grained distinctions, while CoT-based models improve discrimination through explicit generation impractical to serve online. We present Douyin Multimodal Embedding (DME), a model trained in two stages to combine both strengths. Stage 1 performs large-scale contrastive pre-training that establishes a unified multimodal embedding space with broad modality and task coverage. Stage 2 supplements semantic sufficiency, the property that an embedding is grounded in retrieval-relevant evidence and preserves fine-grained counterpart-side semantics, via two mechanisms. Evidence-Grounded Typed Latent Reasoning organizes retrieval evidence through hidden-space latent reasoning, and Cross-Conditional Reconstruction enforces counterpart-side semantics through cross-directional autoregressive reconstruction. Both act only during training and add only marginal query-side overhead, so DME serves as efficiently as a standard contrastive encoder. On MMEB-v2, DME reaches state-of-the-art results at comparable scales for its 2B and 9B variants (74.8 and 78.4), with especially strong video and visual-document tasks. In production, DME delivers a 2.92% relative gain on Douyin's in-house offline evaluation set, is deployed across Douyin scenarios such as generative, image, and AI search, and yields a 0.1% Lifetime (LT) gain in online A/B testing on Douyin search.