理解协同过滤中的嵌入缩放

摘要

將推薦模型擴展為大型推薦模型已成為最廣泛討論的話題之一。近期的研究重點已轉向超越嵌入維度擴展的組件，因為人們認為單純擴展嵌入維度可能導致性能下降。儘管對嵌入已有一些初步觀察，但其不可擴展性的根本原因仍不明確。此外，性能下降是否在不同類型的模型和數據集中普遍存在，仍是一個未經探索的領域。針對嵌入維度對性能的影響，我們在10個稀疏程度和規模各異的數據集上，使用4種代表性的經典架構進行了大規模實驗。我們意外地觀察到兩種新現象：雙峰現象和對數現象。對於前者，隨著嵌入維度的增加，性能先提升後下降，再次上升，最終又下降；對於後者，則呈現出完美的對數曲線。我們的研究貢獻有三方面：首先，我們發現了在擴展協同過濾模型時的兩種新現象；其次，我們深入理解了雙峰現象的成因；最後，我們從理論上分析了協同過濾模型的噪聲魯棒性，其結果與實證觀察相符。

English

Scaling recommendation models into large recommendation models has become one of the most widely discussed topics. Recent efforts focus on components beyond the scaling embedding dimension, as it is believed that scaling embedding may lead to performance degradation. Although there have been some initial observations on embedding, the root cause of their non-scalability remains unclear. Moreover, whether performance degradation occurs across different types of models and datasets is still an unexplored area. Regarding the effect of embedding dimensions on performance, we conduct large-scale experiments across 10 datasets with varying sparsity levels and scales, using 4 representative classical architectures. We surprisingly observe two novel phenomenon: double-peak and logarithmic. For the former, as the embedding dimension increases, performance first improves, then declines, rises again, and eventually drops. For the latter, it exhibits a perfect logarithmic curve. Our contributions are threefold. First, we discover two novel phenomena when scaling collaborative filtering models. Second, we gain an understanding of the underlying causes of the double-peak phenomenon. Lastly, we theoretically analyze the noise robustness of collaborative filtering models, with results matching empirical observations.

理解协同过滤中的嵌入缩放

Understanding Embedding Scaling in Collaborative Filtering

摘要

Support