Echoverse：用于大规模训练计算机使用智能体的深度演化环境

摘要

计算机使用智能体从自身行为所引发的改变中学习，因此训练这样的智能体需要可供其操作、破坏并重置的应用程序。其中最重要的应用往往需要登录且带有状态，于是合成环境便充当其替代品。近期的流水线可以批量生成此类环境，这使瓶颈从环境数量的多少转变为每个环境内部包含的内容。我们发现，回报来自三个特性：环境具备多少行为深度、是否针对智能体实际失败的交互，以及是否随模型一同改进。我们提出了 Echoverse，它将规格说明编译成有状态应用程序，其任务根据应用程序自身的数据库进行评分；同时提出一个协同进化循环，将每条带评分的轨迹读取两次：一次作为修复环境、任务及其验证器的依据，一次作为模型的训练信号。在十二个此类环境上训练后，一个 9B 模型在十四个评估划分上从 36.5% 提升至 67.1%，与训练它的、规模大得多的前沿模型相差在十四个百分点以内。我们依次考察了这些特性。在相同领域中，浅层环境把实际站点准确率压低到基础模型之下（从 80.0 降至 75.0），而深层环境则将其提升（80.0 升至 85.0，48.0 升至 65.0）；对单个界面控件进行跨多种渲染的反复训练，能够迁移到留出的控件族和开放网络上；修复单个环境使基于该环境训练的模型从 16.2% 提升至 38.5%。这些环境同样可用作强化学习环境，其中，将基于数据库的验证器与稠密的逐步骤评判器相结合的奖励函数，把留出集得分从 58.8% 提升至 68.0%。我们发布了四个环境作为基准，包含其应用程序、种子数据和基于数据库的评分器。代码：https://aka.ms/echoverse

English

Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset. The applications that matter most are login-gated and stateful, so synthetic environments stand in for them. Recent pipelines generate such environments in bulk, which moves the bottleneck from how many exist to what is inside each one. The returns, we find, come from three properties: how much behavioural depth an environment carries, whether it targets the interaction an agent actually fails, and whether it improves alongside the model. We present Echoverse, which compiles specifications into stateful applications whose tasks are graded against the application's own database, and a co-evolution loop that reads every graded rollout twice: as repairs to the environment, its tasks and its verifier, and as training signal for the model. Trained on twelve such environments, a 9B model improves from 36.5% to 67.1% across fourteen evaluation splits, within fourteen points of the much larger frontier model that taught it. We examine each property in turn. On the same domains, shallow environments push live-site accuracy below the base model (80.0 to 75.0) while deep ones raise it (80.0 to 85.0 and 48.0 to 65.0); drilling one interface control across many renderings transfers to held-out widget families and to the open web; and repairing a single environment lifts the model trained on it from 16.2% to 38.5%. The same worlds serve as reinforcement-learning environments, where a reward combining the grounded verifier with a dense per-step judge raises held-out score from 58.8% to 68.0%. We release four environments as a benchmark, with their applications, seed data and grounded graders. Code: https://aka.ms/echoverse