OpenForgeRL：在任意環境中訓練原生框架智能體

摘要

現代AI代理依賴如Claude Code、Codex和OpenClaw等複雜的推理框架來驅動多輪推理、工具使用以及外部系統的存取。儘管功能強大，這些複雜的框架也使得代理難以在開放基礎設施上進行端到端訓練，因為其SFT/RL堆疊無法原生表達具狀態、多流程的框架推理。為了解決這個問題，我們提出了OpenForgeRL，一個開源框架，用於在多樣化環境中對基於框架的代理進行端到端訓練。OpenForgeRL透過一個輕量級代理來實現此目標——該代理為框架的模型呼叫提供服務，同時將其記錄為標準RL程式碼庫（例如veRL）的訓練資料，並搭配一個Kubernetes編排器，讓每次執行軌跡都在各自的遠端容器中運行，從而能在任何環境中對任何框架進行大規模訓練。透過將訓練與推理解耦，OpenForgeRL讓研究者能夠直接在代理實際部署的真實框架與環境中，輕鬆訓練、研究並改進代理。我們在多種複雜的框架與環境中驗證了我們的框架，涵蓋基於工具/爪類的代理，以及多模態GUI瀏覽器與電腦使用代理。僅使用數百到數千個任務，OpenForgeClaw在ClawEval上達到了31.7 pass^3與55.9 pass@3，在QwenClawBench上達到了33.7。OpenForgeGUI在OSWorld-Verified上達到37.7，在Online-Mind2Web上達到63.0，在WebVoyager上達到72.3。兩者在幾乎所有基準測試中都優於相似大小的開放基線，且在GUI設定中達到或超越數倍規模更大的模型。除了基準測試之外，我們也分析了框架選擇（例如ZeroClaw、OpenClaw、Codex）與強化學習如何塑造代理行為。我們發現某些框架比起其他框架難學很多，而且強化學習提升了代理的可靠性，例如自我驗證、工具涵蓋範圍以及完成多步驟計畫，但關鍵能力如錯誤恢復仍然薄弱。

English

Modern AI agents rely on elaborate inference harnesses such as Claude Code, Codex, and OpenClaw to drive multi-turn reasoning, tool use, and access to external systems. While powerful, these complex harnesses also make agents hard to train end-to-end with open infrastructure, whose SFT/RL stacks cannot natively express stateful, multi-process harness inference. To address this, we present OpenForgeRL, an open-source framework for training harness-based agents end-to-end in diverse environments. OpenForgeRL achieves this with a lightweight proxy that serves the harness's model calls while recording them as training data for a standard RL codebase (e.g., veRL), and a Kubernetes orchestrator that runs each rollout in its own remote container, together enabling training on any harness in any environment at scale. By decoupling training and inference, OpenForgeRL allows researchers to easily train, study, and improve agents directly in the real harnesses and environments they are deployed with. We validate our framework across diverse, complex harnesses and environments, spanning tool/claw-based agents and multimodal GUI browser- and computer-use agents. Using only hundreds to a few thousand tasks, OpenForgeClaw reaches 31.7 pass^3 and 55.9 pass@3 on ClawEval and 33.7 on QwenClawBench. OpenForgeGUI reaches 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager. Both outperform open baselines of similar size on nearly all benchmarks, and in the GUI setting match or surpass models several times larger. Beyond benchmarks, we analyze how harness choice (e.g., ZeroClaw, OpenClaw, Codex) and RL shape agent behavior. We find that some harnesses are substantially harder to learn than others, and that RL improves agentic reliability, such as self-verification, tool coverage, and completing multi-step plans, though critical abilities such as error recovery remain weak.