邁向十億級規模下用戶表徵學習的密集化定律

摘要

在真實工業場景中，用戶表示學習通常透過擴大用戶數量、行為序列長度與模型規模來進行擴展。然而，現有方法面臨兩項挑戰：（i）在十億級規模下，原始資料擴展存在瓶頸——當原始文本用戶行為輸入的規模擴大時，效能增益會遞減，而詞元化可緩解此問題；（ii）缺乏對詞元化配置應如何隨資料規模擴展的定量分析。本報告提出「用戶行為緻密化定律」，用於刻畫資料規模與最小足夠詞元化容量之間的定量關係。首先，我們在十億級支付寶資料集上進行原始資料與詞元化擴展的對比先導研究，揭示原始資料擴展的瓶頸以及詞元化帶來的持續增益。為推導在不同資料規模下支配最小足夠詞元化配置的擴展規律，我們結合理論分析與系統性實驗，總結出定量擴展模式。我們發現，最小足夠詞元化容量與以詞元計量的輸入資料規模，兩者的對數之間存在近似線性關係，且擴展斜率隨詞元化方法與資料來源系統性地變化，反映表示空間冗餘度與來源內部獨特性的差異。在上述定律的指引下，我們進一步開發 ALGN——一種自適應可變長度詞元化方法，以改善容量配置。涵蓋多種資料來源、詞元化方法與下游任務的大量實驗證實了「用戶行為緻密化定律」的泛化性與可靠性，為大規模用戶表示學習中的詞元化配置選擇提供實用指引。此外，ALGN 優於現有基線方法。

English

User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.