ClawGym II：於智能體測試框架上探索黑盒強化學習

摘要

Agent 框架（agent harnesses）透過協調代理與環境之間的互動，已大幅提升了長時程任務的表現。然而，透過複雜框架進行強化學習仍 largely 未獲探索，因為將此類訓練擴展至長時程代理任務會帶來根本性的挑戰。在本工作中，我們提出了一個統一的黑箱強化學習框架，用於透過複雜框架對通用代理進行穩定且可擴展的最佳化。具體而言，我們首先建構了一個基於沙箱的執行基礎設施，將任務環境與框架隔離於臨時沙箱中，以支援大規模並行軌跡生成。接著，我們將策略最佳化與不透明的框架執行進行解耦，並在模型邊界部署服務代理（serving proxy）以擷取模型呼叫。為了重建多輪軌跡並提升訓練效率，我們將擷取到的呼叫組織為前綴樹，並進一步調整基於評論家的 PPO 與無評論家的 GRPO，以在還原出的樹狀結構上進行最佳化。同時，我們在整個最佳化過程中維持訓練與推論之間的一致性。最後，我們引入了混合框架訓練（mix-harness training），使單一模型能夠透過異質框架進行聯合最佳化。採用 Qwen3-30A3B 時，黑箱強化學習在 ClawGym-Bench 上分別透過 OpenClaw 與 Claude Code 將 Pass@1 提升了 9.98 與 14.81 個百分點，且在 200–400 步最佳化過程中保持穩定。此外，該框架在更具挑戰性的任務（如 JobBench 與 OfficeQA）上也展現了持續的效能提升。整體而言，我們的框架實現了透過黑箱框架對通用代理進行有效、穩定且可擴展的最佳化，並支援跨異質執行系統的統一訓練。

English

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.