LLM2Vec-Gen：基於大型語言模型的生成式嵌入技術

LLM2Vec-Gen: Generative Embeddings from Large Language Models

March 11, 2026

作者: Parishad BehnamGhader, Vaibhav Adlakha, Fabian David Schmidt, Nicolas Chapados, Marius Mosbach, Siva Reddy

cs.AI

摘要

基於大型語言模型的文字嵌入器通常會對輸入語義內容進行編碼。然而，嵌入任務需要將多樣化的輸入映射到相似的輸出。傳統上，這種輸入輸出映射是通過對比學習使用配對數據訓練嵌入模型來實現的。本研究提出一種新穎的自監督方法LLM2Vec-Gen，採用不同範式：我們不直接編碼輸入內容，而是學習表徵模型潛在的響應。具體而言，我們在LLM詞彙表中添加可訓練的特殊標記，將其附加至輸入後端，並優化這些標記以固定長度序列表徵LLM的響應。訓練過程由LLM自身對查詢的補全結果引導，同時結合提供蒸餾目標的無監督嵌入教師模型。此設計有助於彌合輸入輸出間的差距，並將LLM的安全對齊與推理等能力遷移至嵌入任務。關鍵在於，LLM骨幹網絡保持凍結狀態，且訓練僅需未標註的查詢數據。LLM2Vec-Gen在Massive Text Embedding Benchmark（MTEB）上實現了最先進的自監督性能，相較最佳無監督嵌入教師模型提升9.3%。我們還觀察到嵌入任務中有害內容檢索量減少達43.2%，推理能力提升29.3%。最終，學習得到的嵌入具備可解釋性，可通過解碼為文字揭示其語義內容。

English

LLM-based text embedders typically encode the semantic content of their input. However, embedding tasks require mapping diverse inputs to similar outputs. Typically, this input-output is addressed by training embedding models with paired data using contrastive learning. In this work, we propose a novel self-supervised approach, LLM2Vec-Gen, which adopts a different paradigm: rather than encoding the input, we learn to represent the model's potential response. Specifically, we add trainable special tokens to the LLM's vocabulary, append them to input, and optimize them to represent the LLM's response in a fixed-length sequence. Training is guided by the LLM's own completion for the query, along with an unsupervised embedding teacher that provides distillation targets. This formulation helps to bridge the input-output gap and transfers LLM capabilities such as safety alignment and reasoning to embedding tasks. Crucially, the LLM backbone remains frozen and training requires only unlabeled queries. LLM2Vec-Gen achieves state-of-the-art self-supervised performance on the Massive Text Embedding Benchmark (MTEB), improving by 9.3% over the best unsupervised embedding teacher. We also observe up to 43.2% reduction in harmful content retrieval and 29.3% improvement in reasoning capabilities for embedding tasks. Finally, the learned embeddings are interpretable and can be decoded into text to reveal their semantic content.