RexBERT：面向电子商务领域的语境专用双向编码器

RexBERT: Context Specialized Bidirectional Encoders for E-commerce

February 4, 2026

作者: Rahul Bajaj, Anuj Garg

cs.AI

摘要

在检索、分类和排序等对延迟性、稳定性和成本要求极高的系统中，仅编码器架构的Transformer模型仍具有不可替代的价值。然而，大多数通用编码器仅基于覆盖专业领域有限的通用语料库进行训练。我们推出RexBERT系列——专为电子商务语义设计的BERT风格编码器模型，并作出三项贡献：首先，我们发布Ecom-niverse语料库，这是一个从多元零售与购物资源中精选构建的3500亿词元数据集。我们提出模块化流水线方案，能够从FineFineWeb等开放网络资源中隔离提取电商内容，并对最终领域分布特征进行量化分析。其次，我们基于ModernBERT的架构创新提出可复现的预训练方案。该方案包含三阶段训练流程：通用预训练、上下文扩展及退火式领域专项优化。第三，我们训练了参数量从1700万到4亿不等的RexBERT模型，并基于电商数据集在词元分类、语义相似度及通用自然语言理解任务上进行评估。实验表明，尽管参数量减少2-3倍，RexBERT在领域特定基准测试中不仅超越更大规模的通用编码器，更能媲美甚至优于现代长上下文模型。我们的研究证明，高质量领域内数据与原则性训练方法的结合，能为电商应用提供比盲目扩大模型规模更坚实的基础。

English

Encoder-only transformers remain indispensable in retrieval, classification, and ranking systems where latency, stability, and cost are paramount. Most general purpose encoders, however, are trained on generic corpora with limited coverage of specialized domains. We introduce RexBERT, a family of BERT-style encoders designed specifically for e-commerce semantics. We make three contributions. First, we release Ecom-niverse, a 350 billion token corpus curated from diverse retail and shopping sources. We describe a modular pipeline that isolates and extracts e-commerce content from FineFineWeb and other open web resources, and characterize the resulting domain distribution. Second, we present a reproducible pretraining recipe building on ModernBERT's architectural advances. The recipe consists of three phases: general pre-training, context extension, and annealed domain specialization. Third, we train RexBERT models ranging from 17M to 400M parameters and evaluate them on token classification, semantic similarity, and general natural language understanding tasks using e-commerce datasets. Despite having 2-3x fewer parameters, RexBERT outperforms larger general-purpose encoders and matches or surpasses modern long-context models on domain-specific benchmarks. Our results demonstrate that high quality in-domain data combined with a principled training approach provides a stronger foundation for e-commerce applications than indiscriminate scaling alone.