用户-LLM：利用用户嵌入实现高效的LLM语境化

User-LLM: Efficient LLM Contextualization with User Embeddings

February 21, 2024

作者: Lin Ning, Luyang Liu, Jiaxing Wu, Neo Wu, Devora Berlowitz, Sushant Prakash, Bradley Green, Shawn O'Banion, Jun Xie

cs.AI

摘要

大型语言模型（LLMs）已经彻底改变了自然语言处理。然而，有效地整合复杂且可能带有噪音的用户交互数据仍然是一个挑战。为了解决这个问题，我们提出了User-LLM，这是一个新颖的框架，利用用户嵌入来对LLMs进行上下文化。这些嵌入是通过自监督预训练从各种用户交互中提炼出来的，捕捉了潜在的用户偏好以及随时间演变的情况。我们通过交叉注意力和软提示将这些用户嵌入与LLMs整合在一起，使LLMs能够动态地适应用户上下文。我们在MovieLens、Amazon Review和Google Local Review数据集上进行了全面的实验，展示了在各种任务中显著的性能提升。值得注意的是，我们的方法在长序列任务和需要深入理解用户的任务上优于基于文本提示的上下文化方法，同时具有高效的计算性能。我们进一步将Perceiver层整合到用户编码器和LLMs之间，以简化整合过程，降低计算需求。

English

Large language models (LLMs) have revolutionized natural language processing. However, effectively incorporating complex and potentially noisy user interaction data remains a challenge. To address this, we propose User-LLM, a novel framework that leverages user embeddings to contextualize LLMs. These embeddings, distilled from diverse user interactions using self-supervised pretraining, capture latent user preferences and their evolution over time. We integrate these user embeddings with LLMs through cross-attention and soft-prompting, enabling LLMs to dynamically adapt to user context. Our comprehensive experiments on MovieLens, Amazon Review, and Google Local Review datasets demonstrate significant performance gains across various tasks. Notably, our approach outperforms text-prompt-based contextualization on long sequence tasks and tasks that require deep user understanding while being computationally efficient. We further incorporate Perceiver layers to streamline the integration between user encoders and LLMs, reducing computational demands.