SIRI：通过交错压缩实现迭代强化学习的规模化扩展

SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression

September 29, 2025

作者: Haoming Wen, Yushi Bai, Juanzi Li, Jie Tang

cs.AI

摘要

我们提出SIRI（Scaling Iterative Reinforcement Learning with Interleaved Compression），一种针对大型推理模型（LRMs）的简单而有效的强化学习方法，旨在实现更高效、更准确的推理。现有研究已观察到LRMs中存在重复的思维模式，而减少这些重复的尝试往往以性能下降为代价。本文中，我们展示了一种通过训练机制克服这一权衡的方法，即在训练过程中动态调整最大展开长度，交替进行压缩与扩展。压缩阶段削减展开长度，迫使模型在有限上下文中做出精确且有价值的决策，从而有效减少冗余标记并提高推理密度。扩展阶段则放宽长度限制，为模型在长视野场景中探索与规划提供空间。值得注意的是，我们发现每经过一次压缩-扩展循环，模型的性能都会提升，即使其输出长度减少，稳步推动其接近性能-效率权衡的帕累托前沿。在DeepSeek-R1-Distill-Qwen-1.5B上训练后，SIRI-low在AIME24上的性能提升了43.2%，同时标记使用量减少了46.9%，经过三次迭代；而SIRI-high相比所有其他方法达到了最高准确率（图1）。我们的发现揭示了在训练期间周期性调整LRM输出截断长度以动态平衡推理探索与效率的潜力，最终在两者之间达到一个最优的“甜蜜点”。我们的模型已公开可用。

English

We introduce SIRI, Scaling Iterative Reinforcement Learning with Interleaved Compression, a simple yet effective RL approach for Large Reasoning Models (LRMs) that enables more efficient and accurate reasoning. Existing studies have observed repetitive thinking patterns in LRMs, and attempts to reduce them often come at the cost of performance. In this paper, we show that this trade-off can be overcome through a training regime that iteratively alternates between compressing and expanding the reasoning budget, by dynamically adjusting the maximum rollout length during training. The compression phase cuts the rollout length, forcing the model to make precise and valuable decisions within a limited context, which effectively reduces redundant tokens and increases reasoning density. The expansion phase then relaxes the length limit, providing space for the model to explore and plan in long-horizon settings. Remarkably, we find that after each compression-expansion cycle, the model's performance improves even as its output length decreases, steadily pushing it closer to the Pareto frontier in the performance-efficiency trade-off. Training on DeepSeek-R1-Distill-Qwen-1.5B, SIRI-low improves performance on AIME24 by 43.2% while reducing token usage by 46.9% after three iterations, and SIRI-high achieves the highest accuracy compared to all other methods (Figure 1). Our findings shed light on the potential of periodically oscillating the LRM's output truncation length during training to dynamically balance exploration and efficiency in reasoning, converging towards an optimal "sweet spot" between the two. Our models are publicly available.