Ring-Zero：将零强化学习扩展至万亿参数以实现涌现推理

摘要

无需人工标注数据的可验证奖励强化学习（常被称为零强化学习）已成为激发思维链推理的强大范式。然而，受限于计算资源，现有研究多局限于小规模模型，大规模模型的训练动态与涌现能力尚未得到充分探索。为切实推进这一前沿领域，我们致力于引导模型生成高质量的推理行为。然而，我们发现简单的规模化扩展往往面临可读性差、标记冗余以及缺乏自适应推理深度等问题。针对这些挑战，我们提出了一种稳定高效的训练流程，集成了算法与系统层面的优化手段，包括裁剪重要性采样、训练-推理比例校正以及混合精度控制。我们的实验揭示了三个关键发现，验证了规模化扩展的“苦涩教训”：（1）将模型扩展至1万亿参数可显著提升样本效率与性能上限；（2）训练过程依次经历初始发现阶段与锐化阶段；（3）模型自发涌现出高级认知行为，包括拟人化、结构化格式、自我验证、并行推理与情境焦虑，使得人工设计的启发式策略变得多余。在七个数学基准测试上，Ring-2.5-1T-Zero取得了具有竞争力的表现。此外，为超越最终答案正确性来评估思维链质量，我们提出了一个涵盖三个维度的结构化评估框架：可理解性、可复现性与效率。在该框架下，我们的模型在生成结构化且简洁的推理轨迹方面展现出明显优势。通过分享我们观察到的涌现现象，我们希望为社区提供关于规模化行为（尤其是在万亿参数规模下）的更深入洞察。

English

Reinforcement learning with verifiable rewards without human-annotated data, often referred to as zero RL, has emerged as a powerful paradigm for eliciting chain-of-thought reasoning. However, due to computational constraints, existing studies are largely restricted to small models, leaving the training dynamics and emergent capabilities at a large scale unexplored. To meaningfully explore this frontier, we aim to elicit high-quality reasoning behaviors from the model. However, we find that naive scaling often suffers from poor readability, token redundancy, and a lack of adaptive reasoning depth. To address these challenges, we present a stable and efficient training pipeline, incorporating algorithmic and system optimizations such as clipped importance sampling, training-inference ratio correction, and mixed-precision control. Our experiments offer three key findings that validate the "bitter lesson" of scaling: (1) scaling to 1T parameters significantly enhances sample efficiency and performance ceilings; (2) the training process progresses sequentially through an initial discovery phase followed by a sharpening phase; and (3) the model spontaneously develops advanced cognitive behaviors, including anthropomorphism, structured formatting, self-verification, parallel reasoning, and context anxiety, rendering hand-crafted heuristics redundant. Evaluated on seven mathematical benchmarks, Ring-2.5-1T-Zero achieves competitive performance. Additionally, to assess CoT quality beyond final-answer correctness, we propose a structured evaluation framework across three dimensions: comprehensibility, reproducibility, and efficiency, where our model demonstrates clear advantages in producing structured and concise reasoning traces. By sharing our observed emergent phenomena, we hope to provide the community with deeper insights into scaling behaviors, particularly at the 1-trillion scale.

Ring-Zero：将零强化学习扩展至万亿参数以实现涌现推理

Ring-Zero: Scaling Zero RL to a Trillion Parameters for Emergent Reasoning

摘要

Support