EnvACE: 通过世界预演内化环境动态的智能体强化学习

摘要

训练大型语言模型智能体进行长时程工具使用，通常依赖于与真实或合成可执行环境的交互，而这些环境的构建与验证成本高昂，或者依赖难以接地（ground）的外部模拟器。我们提出EnvACE，一种智能体强化学习方法，在训练期间用世界预演（world rehearsal）替代外部环境交互。策略在行动与预演之间交替：它首先生成一个工具调用，然后扮演环境角色，产生由该动作引发的响应，并基于预演的响应来调节后续决策。两个角色通过任务成功奖励进行端到端联合优化。通过世界预演，策略将动作与其环境响应之间的关系内化到参数中，形成一个直接支持决策的智能体世界模型。在BFCL-v4、tau^2-Bench、VitaBench和FinMCP-Bench上，EnvACE实现了强大且可迁移的性能，在整体评估中优于环境扩展基线。对照研究进一步表明，世界预演在不同模型规模上持续改善策略学习。在测试时，内化的世界模型能够在承诺执行之前进行私有预演，在适度的预演预算下，无需额外外部交互即可获得进一步增益。我们的发现确立了世界预演作为一条超越外部环境约束、扩展LLM智能体训练的新路径。我们的代码公开于https://github.com/Within-yao/EnvACE。

English

Training large language model agents for long-horizon tool use typically relies on interactions with real or synthesized executable environments, whose construction and verification are costly, or on external simulators that are difficult to ground. We introduce EnvACE, an agentic reinforcement learning method that replaces external environment interaction during training with world rehearsal. The policy alternates between acting and rehearsal: it first generates a tool call, then plays the role of the environment to produce the response induced by that action, and conditions subsequent decisions on the rehearsed response. Both roles are jointly optimized end-to-end using task-success rewards. Through world rehearsal, the policy internalizes the relationship between actions and their environment responses in its parameters, yielding an agent world model that directly supports decision making. Across BFCL-v4, tau^2-Bench, VitaBench, and FinMCP-Bench, EnvACE achieves strong and transferable performance, outperforming environment-scaling baselines in the overall evaluation. Controlled studies further show that world rehearsal consistently improves policy learning across model scales. At test time, the internalized world model enables private rehearsal before committed execution, yielding further gains under a moderate rehearsal budget without additional external interaction. Our findings establish world rehearsal as a new path toward scaling LLM agent training beyond the constraints of external environments. Our code is publicly available at https://github.com/Within-yao/EnvACE.