SPADE：自適應性合成可執行環境中的自我對弈

摘要

持續自我改進需要不斷擴大的自我生成、多樣化且自適應的目標池。對於語言代理而言，現有的訓練環境池（人工精心策劃、靜態合成或凍結驗證器）在學習者規模擴大時，會使目標分布保持固定。我們提出 SPADE（自適應合成可執行環境中的自對弈），這是一個自對弈強化學習框架，其中單一大型語言模型扮演兩個角色：一個環境設計器，負責將完整、長視野的訓練環境編寫為具有 OpenAI Gym 風格 reset()/step() 介面的可執行代碼；以及一個推理代理，負責學習在這些環境中行動。每個環境都是具狀態的多輪環境（包含狀態轉換、獎勵函數和驗證代碼），因此單一介面即可涵蓋推理問題與多步驟代理型工具使用。推理代理的遺憾值透過其在有無特權提示情況下的獎勵差距來估計；在優化此遺憾信號的過程中，環境設計器學會針對代理能力邊緣的環境進行設計，同時保持這些環境的可行性。透過大量實驗，我們發現幾個對成功至關重要的組件：將環境設計器基於從大型預訓練語料庫中採樣的文件進行接地，並賦予其累積的環境記憶。將模型擴展至 300 億參數時，SPADE 在八個留出的數學、科學、代碼和推理基準上，平均比最強的固定環境基線提高 5.3；在工具使用設置上，於 BFCL-v4 多輪基準提升 5.7，於 ACEBench-Agent 提升 13.9；在遊戲設置中，與最強基線的差距隨模型規模增大而擴大。透過使環境設計本身成為可學習的組件，SPADE 向開放式自我改進邁出了具體的一步。

English

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.