LM-词库：通过协调语义专家提升定义建模效能

摘要

我们提出LM-Lexicon——一种融合数据聚类、语义专家学习与稀疏专家混合架构模型融合的创新定义建模方法。该方法通过将定义建模任务分解为特定语义域，并训练小型语言模型作为领域专家，在五个广泛使用的基准测试中相较原有最优模型实现了显著提升（BLEU分数提高7%）。实证研究表明：1）聚类策略可实现细粒度专家 specialization，使定义质量提升近10%；2）语义感知的域级路由机制相较传统词元级路由提升专家效能1%；3）通过测试时计算资源调配与语义专家规模扩展可获得额外性能增益。本研究成果在推动定义建模发展的同时，为语义密集型应用的高效语言模型开发提供了重要洞见。

English

We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications.

LM-词库：通过协调语义专家提升定义建模效能

LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts

摘要

Support