Morpheus：一種針對土耳其語的語形感知神經分詞器與詞嵌入器

摘要

土耳其语是一种黏着语：其意义由词素承载，然而驱动现代语言模型的子词分词器却依据语料统计信息拆分单词，导致承载语义的后缀被碎片化——在WordPiece和基于规则的分析器中，甚至无法将输出还原为原始文本。本文提出Morpheus，一种面向土耳其语的神经词素边界模型，它同时具备无损、形态感知的分词器与词嵌入生成器的功能。一个可微的泊松二项动态规划机制在训练时将每个字符的边界概率转化为软词素隶属度，在推理时生成精确的分段，且无需任何字符串归一化处理，从而在构造上保证decode(encode(w)) = w成立。由于模型是神经网络的，同一前向传播过程既能完成分词，又能输出结构化词嵌入。在可逆分词器（唯一适用于文本生成的类型）中，Morpheus取得了最低的每字符位数（1.425），其形态对齐效果约为子词家族的同类模型的两倍（MorphScore宏F1值为0.61，而后者约为0.32），并且GPU内存使用量比64K词汇量的子词分词器低约19%。作为嵌入器，冻结的Morpheus向量在词汇检索（root-family MAP 0.85）和同根验证（ROC-AUC 1.00）任务上表现领先，超越了多语言检索器BGE-M3和BERTurk；而在依赖上下文和屈折变化的任务（NER、格/数探测）中，更重的上下文编码器依然保持优势——我们将这一权衡归因于Morpheus以词根为中心的几何结构。代码：https://github.com/lonewolf-rd/TurkishMorpheus；模型：https://huggingface.co/lonewolflab/Morpheus-TR-50K；交互式演示：https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo。

English

Turkish is agglutinative: meaning is carried by morphemes, yet the subword tokenizers that drive modern language models split words by corpus statistics, fragmenting semantically loaded suffixes and -- in the case of WordPiece and rule-based analyzers -- failing to decode their output back to the original text. This paper presents Morpheus, a neural morpheme-boundary model for Turkish that is at once a lossless, morphology-aware tokenizer and a word-embedding producer. A differentiable Poisson-binomial dynamic program turns per-character boundary probabilities into soft morpheme memberships during training and exact segments at inference, with no string normalization, so decode(encode(w)) = w holds by construction. Because the model is neural, the same forward pass that tokenizes also emits a structured word embedding. Among reversible tokenizers -- the only ones valid for generation -- Morpheus attains the lowest bits-per-character (1.425), roughly doubles the gold morphological alignment of the subword family (MorphScore macro-F1 0.61 vs.\ {sim}0.32), and uses {sim}19% less GPU memory than 64K-vocabulary subword tokenizers. As an embedder, frozen Morpheus vectors lead on lexical retrieval (root-family MAP 0.85) and same-root verification (ROC-AUC 1.00), surpassing the multilingual retriever BGE-M3 and BERTurk; on context- and inflection-dependent tasks (NER, case/number probing) the heavier contextual encoders remain ahead -- a trade-off we attribute to Morpheus's root-centric geometry. Code: https://github.com/lonewolf-rd/TurkishMorpheus; model: https://huggingface.co/lonewolflab/Morpheus-TR-50K; interactive demo: https://huggingface.co/spaces/lonewolflab/morpheus-tr-demo.