何處切分，多深為宜：化學SMILES中的BPE與Unigram-LM

摘要

每一个读取SMILES的化学语言模型都始于分词器，然而该领域几乎未经审视地继承了自然语言处理中的字节对编码（BPE）。在自然语言中，BPE的主要替代方案——Unigram-LM——已知会构建结构不同的词汇表。这种差异是否存在于化学领域尚不明确。我们报告了在固定165词元化学基组上、于词元嵌入可学习的小词汇量规模下，对BPE与Unigram-LM进行的受控对比研究，覆盖三种语料类型（多样性、类药、天然产物）及两种预分词边界策略。两者并未收敛。在所有22组匹配条件下，它们构建了近乎不重合的子词词汇表：跨算法下，学习到的词片段的杰卡德重叠系数从未超过0.161；若加权偏向模型更新最频繁的高频片段，则至多为0.05。Unigram-LM还将保留分子分割成29%–41%更多的词元；两种方法在何处切分上基本一致，但在切分深度上不同——在80%–99%的分子上，BPE的分割是Unigram-LM的严格粗化版本。这种分离在语料、边界和词汇量维度上均成立，甚至在规模扩大八倍时依然存在。因此，子词算法是一项建模决策，而非默认选项。本研究未训练任何语言模型。

English

Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.