迈向十亿级容量下用户表征学习的致密化定律

摘要

用户表示学习在真实工业场景中的扩展通常通过增加用户数量、行为序列长度和模型规模来实现。然而，现有方法面临两大挑战：(i) 十亿级容量规模下原始数据扩展的瓶颈，因为更大规模的原始文本用户行为输入所对应的性能增益逐渐递减，而这一瓶颈可以通过词元化来缓解；(ii) 缺乏关于词元化配置应如何随数据规模扩展的定量分析。在本报告中，我们提出用户行为稠密化定律，用于刻画数据规模与最小充分词元化容量之间的定量关系。首先，我们在十亿级支付宝数据集上开展了原始数据与词元化数据扩展对比的初步研究，揭示了原始数据扩展的瓶颈以及词元化所带来的持续增益。为推导不同数据规模下支配最小充分词元化配置的扩展规律，我们结合理论分析和系统性实验，总结了相应的定量扩展模式。我们发现，最小充分词元化容量的对数与以词元数衡量的输入数据规模的对数之间近似呈线性关系，且扩展斜率随词元化方法和数据源的不同而系统性变化，反映了表示空间冗余和源内唯一性的差异。在该定律的指导下，我们进一步提出了 ALGN——一种自适应变长词元化方法，可改善容量分配。跨多种数据源、词元化方法和下游任务的大规模实验证明了用户行为稠密化定律的泛化性和可靠性，为大规模用户表示学习中的词元化配置选择提供了实用指导。此外，ALGN 的性能优于现有基线方法。

English

User representation learning in real-world industrial scenarios is commonly scaled by increasing user amount, behavioral sequence length and model size. However, existing methods face two challenges: (i) Bottleneck for raw data scaling at billion-scale capacity, as performance exhibit diminishing performance gains with larger-scale raw text user behavioral input, which can be mitigated by tokenization. (ii) Lack of quantitative analysis of how tokenization configurations should scale with data size. In this report, we propose User Behavioral Densing Law for characterizing the quantitative relationship between data scale and the minimum sufficient tokenization capacity. Firstly, we conduct a pilot study on raw & tokenized scaling comparison on billion-scale Alipay dataset, revealing the raw data scaling bottleneck and the sustained gains enabled by tokenization. To derive the scaling pattern governing the minimum sufficient tokenization configuration at different data scales, theoretical analysis and systematic experiments are employed to summarize the quantitative scaling pattern. We find an approximately linear relationship between the logarithms of minimum sufficient tokenization capacity and input data size measured by tokens, and the scaling slope varies systematically with the tokenization method and data source, reflecting differences in representation-space redundancy and intra-source uniqueness. Guided by the proposed law, we further develop ALGN, an adaptive variable-length tokenization method that improves capacity allocation. Extensive experiments across diverse data sources, tokenization methods, and downstream tasks demonstrate the generalizability and reliability of the User Behavioral Densing Law, providing practical guidance for tokenization configuration selection in large-scale user representation learning. Moreover, ALGN outperforms existing baselines.