ClawGym II：基于智能体框架的黑盒强化学习探索

摘要

智能体框架（Agent harnesses）通过协调智能体与环境的交互，已显著提升了长时域任务的性能。然而，通过复杂框架进行强化学习仍 largely 未被探索，因为将此类训练扩展到长时域智能体任务会引入根本性挑战。在本研究中，我们提出了一种统一的黑盒强化学习框架，用于通过复杂框架对通用智能体进行稳定且可扩展的优化。具体而言，我们首先构建了一个基于沙箱的执行基础设施，将任务环境和框架隔离在临时沙箱中，以支持大规模并发 rollout。随后，我们将策略优化与不透明的框架执行解耦，并在模型边界处部署服务代理以捕获模型调用。为重构多轮轨迹并提升训练效率，我们将捕获的调用组织成前缀树，并进一步适配基于评论家的 PPO 和免评论家的 GRPO 算法，以在恢复的树结构上进行优化。同时，我们在整个优化过程中保持训练与推理的一致性。最后，我们引入了混合框架训练（mix-harness training），使单一模型能够通过异构框架进行联合优化。基于 Qwen3-30A3B，黑盒强化学习在 ClawGym-Bench 上通过 OpenClaw 和 Claude Code 分别将 Pass@1 提升了 9.98 和 14.81 个百分点，且在 200–400 步优化过程中保持稳定。此外，该框架在更具挑战性的任务（如 JobBench 和 OfficeQA）上也取得了持续的性能提升。总体而言，我们的框架通过黑盒框架实现了对通用智能体的有效、稳定且可扩展的优化，并支持跨异构执行系统的统一训练。

English

Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment. However, reinforcement learning through complex harnesses remains largely unexplored, as scaling such training to long-horizon agent tasks introduces fundamental challenges. In this work, we present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses. Concretely, we first build a sandbox-based execution infrastructure that isolates task environments and harnesses within temporary sandboxes for large-scale concurrent rollouts. We then decouple policy optimization from opaque harness execution and place a serving proxy at the model boundary to capture model calls. To reconstruct multi-turn trajectories and improve training efficiency, we organize the captured calls into prefix trees and further adapt both critic-based PPO and critic-free GRPO to optimize over the recovered tree structure. Meanwhile, we maintain training-inference consistency throughout the optimization process. Finally, we introduce mix-harness training, allowing a single model to be jointly optimized by heterogeneous harnesses. With Qwen3-30A3B, black-box RL improves Pass@1 on ClawGym-Bench by 9.98 and 14.81 points through OpenClaw and Claude Code, respectively, while remaining stable over 200-400 optimization steps. Moreover, the framework yields consistent gains on more challenging tasks such as JobBench and OfficeQA. Overall, our framework enables effective, stable, and scalable optimization of general agents through black-box harnesses, supporting unified training across heterogeneous execution systems.