SPADE：自适应合成可执行环境中的自我对弈

摘要

持续自我提升需要不断扩充由自我生成的、多样化的、适应性强的目标池。对于语言智能体而言，现有的训练环境池（人工精心设计的、静态合成的或固定验证器生成的）会使目标分布在学习者规模扩展时保持固定不变。我们提出 SPADE（自适应合成可执行环境中的自我对弈，Self-Play in Adaptive Synthetic Executable Environments），这是一种自我对弈强化学习框架，其中单个大语言模型扮演两个角色：一个环境设计者（Environment Designer），负责将完整的、长时程训练环境编写为带有 OpenAI Gym 风格 reset()/step() 接口的可执行代码；以及一个推理智能体（Reasoning Agent），负责学习在这些环境中行动。每个环境都是有状态的多轮交互环境（包含状态转移、奖励函数和验证代码），因此同一接口既可涵盖推理问题，也可涵盖多步智能体工具使用。推理智能体的遗憾通过其在有无特权提示两种情况下奖励之差来估计；在优化这一遗憾信号的过程中，环境设计者学会将目标锁定在智能体能力边界附近，同时保持这些目标可达成。通过大量实验，我们发现若干对成功至关重要的组件：将环境设计者基于从大规模预训练语料库中采样的文档进行锚定，并为其提供累积的环境记忆。在扩展至 300 亿参数模型后，SPADE 在八个保留的数学、科学、代码和推理基准上的平均性能比最强的固定环境基线高出 +5.3；在工具使用场景中，在 BFCL-v4 多轮任务上提升 +5.7，在 ACEBench-Agent 上提升 +13.9；在游戏场景中，相对于最强基线的优势随模型规模扩大而增长。通过将环境设计本身变为可学习组件，SPADE 向开放式自我提升迈出了切实的一步。

English

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.