SPICE：语料环境中的自我博弈提升推理能力

摘要

自我改进系统需要通过环境交互实现持续适应。我们提出SPICE（语料库环境自博弈）强化学习框架，其核心在于单一模型扮演双重角色：作为挑战者从大型语料库中挖掘文档以生成多样化推理任务，同时作为求解者解决这些任务。通过对抗性动态机制，挑战者在求解者能力边界上构建自动课程，而语料库根基则为持续改进提供了丰富且近乎无穷的外部信号。与现有缺乏根基的自博弈方法相比，SPICE在多个模型系列的数学推理（+8.9%）和通用推理（+9.8%）基准测试中均实现稳定提升。分析表明，文档根基是SPICE持续生成并实现日益复杂目标的关键要素，从而驱动系统的持续自我改进。

English

Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement.

SPICE：语料环境中的自我博弈提升推理能力

SPICE: Self-Play In Corpus Environments Improves Reasoning

摘要

Support