迴聲宇宙：大規模訓練電腦使用代理之深度演化環境

摘要

電腦操作代理從其行動所改變的事物中學習，因此訓練這類代理需要它能操作、破壞並重置的應用程式。最重要的應用程式通常設有登入門檻且具有狀態，因此以合成環境代替。近期的管線能大量產生這類環境，使瓶頸從「環境有多少」轉移到「每個環境內部有什麼」。我們發現，效益來自三個特性：環境承載多少行為深度、是否鎖定代理實際會失敗的互動、以及是否隨模型一同改進。我們提出 Echoverse，它將規格編譯為有狀態的應用程式，其任務會依據應用程式自身的資料庫進行評分；同時提出一個共同演化迴圈，將每個已評分的軌跡讀取兩次：一次作為對環境、其任務及其驗證器的修復，另一次作為模型的訓練訊號。在十二個這類環境上訓練後，一個 9B 模型在十四個評估分割上的表現從 36.5% 提升至 67.1%，與教導它的、規模大得多的前沿模型相差在十四個百分點以內。我們依序檢視每個特性。在相同領域中，淺層環境會將真實網站準確率壓到低於基礎模型（80.0 降至 75.0），而深層環境則能提升它（80.0 升至 85.0，48.0 升至 65.0）；在多種渲染中反覆操練單一介面控制項，可遷移至留出的小工具家族及開放網路；而修復單一環境，能將在其上訓練的模型從 16.2% 提升至 38.5%。這些相同的環境也可用作強化學習環境，其中結合基於資料的驗證器與稠密逐步驟評判器的獎勵，可將留出分數從 58.8% 提升至 68.0%。我們釋出四個環境作為基準，包含其應用程式、種子資料與基於資料的評分器。程式碼：https://aka.ms/echoverse

English

Computer-use agents learn from what their actions change, so training one needs applications it can act on, break and reset. The applications that matter most are login-gated and stateful, so synthetic environments stand in for them. Recent pipelines generate such environments in bulk, which moves the bottleneck from how many exist to what is inside each one. The returns, we find, come from three properties: how much behavioural depth an environment carries, whether it targets the interaction an agent actually fails, and whether it improves alongside the model. We present Echoverse, which compiles specifications into stateful applications whose tasks are graded against the application's own database, and a co-evolution loop that reads every graded rollout twice: as repairs to the environment, its tasks and its verifier, and as training signal for the model. Trained on twelve such environments, a 9B model improves from 36.5% to 67.1% across fourteen evaluation splits, within fourteen points of the much larger frontier model that taught it. We examine each property in turn. On the same domains, shallow environments push live-site accuracy below the base model (80.0 to 75.0) while deep ones raise it (80.0 to 85.0 and 48.0 to 65.0); drilling one interface control across many renderings transfers to held-out widget families and to the open web; and repairing a single environment lifts the model trained on it from 16.2% to 38.5%. The same worlds serve as reinforcement-learning environments, where a reward combining the grounded verifier with a dense per-step judge raises held-out score from 58.8% to 68.0%. We release four environments as a benchmark, with their applications, seed data and grounded graders. Code: https://aka.ms/echoverse