面向德语和英语的主权开源基础模型

摘要

我们提出 Soofi S 30B-A3B，这是一个面向德语和英语的主权级、开源混合专家模型，采用 Mamba-Transformer 混合架构。其混合设计使得每 token 仅激活 30B 参数中的 3B，且推理缓存随上下文增长几乎保持恒定，在长上下文、高并发部署场景下，相比稠密模型具有显著的吞吐量优势。该模型在约 27 万亿 token 上预训练，并特意增加了德语权重，在英德综合基准测试上媲美稠密 14B 至 27B 模型，同时在 17 个开源基座模型中，于两种语言均取得了最佳代码综合得分，并超越了我们比较中的所有欧洲主权基线模型——包括那些活跃参数规模更大的模型。在所有完全开源模型中，Soofi S 取得了最高的英德评测得分，领先于 Olmo 3 32B 和 Apertus 70B。Soofi S 基于德国工业 AI 云实现端到端构建，该云是由德国电信在慕尼黑运营的主权级 HPC 规模 AI 基础设施。Soofi S 将以高度宽松的开源条款发布：包含权重、选定的中间检查点、完整的逐来源数据说明、超参数、训练与评测代码。在来源许可允许的情况下，数据构建工件将以宽松许可证发布；商业授权来源则以汇总统计数据和精确混合比例说明的形式记录。

English

We present Soofi S 30B-A3B, a sovereign, open-source Mixture-of-Experts (MoE) hybrid Mamba Transformer foundation model for German and English. Its hybrid design activates only 3B of 30B parameters per token and keeps the inference cache near-constant as context grows, giving it a decisive throughput advantage over dense models for long-context, high-concurrency deployment. Pretrained on roughly 27 trillion tokens with deliberately up-weighted German, Soofi S matches dense 14 to 27B models on aggregate English and German benchmarks while achieving the best code aggregates in both languages among 17 open base models, and outperforms every European sovereign baseline in our comparison, including ones far larger in active parameters. Among fully open models, Soofi S obtains the highest English and German evaluation scores, ahead of Olmo 3 32B and Apertus 70B. Soofi S was built end-to-end on the German Industrial AI Cloud, a sovereign HPC scale AI infrastructure operated by Deutsche Telekom in Munich. Soofi S will be released under highly permissive, open-access terms: weights, selected intermediate checkpoints, full per-source data accounting, hyperparameters, and training and evaluation code. Where source licenses permit, data-construction artifacts are released under permissive licenses; commercially licensed sources are documented with aggregate statistics and exact mixture accounting.