EnvACE：透過世界演練內化環境動態以實現智能體強化學習

摘要

訓練大型語言模型智能體進行長程工具使用時，通常依賴於與真實或合成的可執行環境互動，這些環境的建構與驗證成本高昂；或者依賴難以落地的外部模擬器。我們提出 EnvACE，一種智能體強化學習方法，以世界預演取代訓練期間的外部環境互動。策略在行動與預演之間交替：首先產生一個工具呼叫，然後扮演環境的角色產生該行動所引發的回應，並以預演的回應作為後續決策的條件。這兩個角色利用任務成功獎勵進行端到端聯合優化。透過世界預演，策略在其參數中內化了行動與環境回應之間的關係，產生一個直接支援決策的智能體世界模型。在 BFCL-v4、tau^2-Bench、VitaBench 和 FinMCP-Bench 上，EnvACE 達到了強大且可遷移的效能，在整體評估中優於環境擴展基準。控制實驗進一步顯示，世界預演在不同模型規模下一致地提升策略學習。在測試時，內化的世界模型能夠在實際執行前進行內部預演，在適度的預演預算下無需額外的外部互動即可帶來進一步的收益。我們的研究結果將世界預演確立為一條超越外部環境限制、擴展大型語言模型智能體訓練的新路徑。我們的程式碼公開於 https://github.com/Within-yao/EnvACE。

English

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.