OpenForgeRL：在任何环境中训练Harness原生智能体

摘要

现代AI代理依赖复杂的推理框架（如Claude Code、Codex和OpenClaw）来支持多轮推理、工具调用及外部系统访问。尽管功能强大，但这些复杂框架使得代理难以在开放基础设施上进行端到端训练——其SFT/RL堆栈无法原生支持有状态、多进程的框架推理。为解决这一问题，我们提出OpenForgeRL——一个开源框架，支持在多样化环境中对基于框架的代理进行端到端训练。OpenForgeRL通过轻量级代理实现这一目标：该代理在服务框架的模型调用时，将调用记录为标准RL代码库（如veRL）的训练数据；同时结合Kubernetes编排器，将每次 rollout 部署至独立远程容器，从而支持在任何环境的海量框架上训练。通过解耦训练与推理，OpenForgeRL使研究者能够直接在代理实际部署的框架与环境中便捷地训练、研究及优化代理。我们在多种复杂框架与环境（涵盖基于工具/爪的代理，以及多模态GUI浏览器和计算机操作代理）中验证了该框架。仅使用数百至数千个任务，OpenForgeClaw在ClawEval上达到31.7 pass^3和55.9 pass@3，在QwenClawBench上达到33.7；OpenForgeGUI在OSWorld-Verified上达到37.7，在Online-Mind2Web上达到63.0，在WebVoyager上达到72.3。两者在几乎所有基准测试中均优于相似规模的开源基线，且在GUI场景中匹配或超越数倍规模的模型。除基准测试外，我们分析了框架选择（如ZeroClaw、OpenClaw、Codex）和强化学习如何塑造代理行为。研究发现，部分框架的学习难度显著高于其他框架，且强化学习提升了代理的可靠性（如自我验证、工具覆盖范围及多步计划完成能力），但错误恢复等关键能力仍显薄弱。

English

Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.