通過字節級模擬解耦子詞分詞對語言模型訓練的益處

摘要

子词分词是现代大型语言模型（LLMs）的重要组成部分，但其对训练效率和模型性能的具体贡献仍未被充分理解。在本研究中，我们通过将子词分词的影响隔离在受控的字节级预训练流程中，从而解耦其作用。我们从样本吞吐量、词汇规模扩展以及子词边界的语言先验等多个维度制定并检验假设。通过在字节级设置中模拟这些效应，我们更深入地理解了子词模型为何优于原始字节模型，并为改进未来字节级和子词模型的预训练提供了见解。具体而言，我们的实验凸显了提高训练吞吐量以及将子词边界作为显式先验或归纳偏置进行整合的关键作用。

English

Subword tokenization is an essential part of modern large language models (LLMs), yet its specific contributions to training efficiency and model performance remain poorly understood. In this work, we decouple the effects of subword tokenization by isolating them within a controlled byte-level pretraining pipeline. We formulate and test hypotheses across various dimensions, including sample throughput, vocabulary scaling, and the linguistic prior of subword boundaries. By simulating these effects in a byte-level setting, we refine our understanding of why subword models outperform raw byte models and offer insights to improve the pretraining of future byte-level and subword models. Specifically, our experiments highlight the critical role of increased training throughput and the integration of subword boundaries as either explicit priors or inductive biases.