Tmax：終端智能體的簡單配方

摘要

終端代理已迅速成為語言模型（LM）最受歡迎的下游應用。儘管其普及程度很高，但學術界對基於強化學習（RL）的此類模型訓練研究相對較少，這可能歸因於困難的基準測試、數據缺乏以及缺乏簡單的基礎配方。我們提出了 Tmax，這是最強大的開源終端代理 RL 配方，將開源數據配方推向前沿。儘管方法簡單，我們的配方僅用 9B 參數就在 Terminal-Bench 2.0 上達到了 27% 的效能，超越了先前工作中規模更大的模型。具體而言，我們使用一種新穎的分類法生成數據，結合了難度控制、角色設定及驗證器多樣性，這使我們能夠低成本地生成大量終端環境，用於 RL 和 SFT 訓練。我們開源了終端數據集，其規模比先前發布的終端代理數據集大 2.5 倍以上。隨後，我們使用 RL 訓練數據訓練開放權重模型，採用純粹基於結果的簡單方法。我們在 https://github.com/hamishivi/tmax 上發布了數據、模型和程式碼，作為未來終端代理開放學術研究的強大基線。

English

Terminal-using agents have quickly become the most popular downstream application of language models (LMs). Despite their prevalence, relatively little academic work has examined RL-based training of these models, likely due to difficult benchmarks, a lack of data, and a lack of simple baseline recipes. We present Tmax, the strongest open RL recipe for terminal agents to date, bringing open data recipes closer to the frontier. While simple, our recipe achieves 27\% on Terminal-Bench 2.0 with only 9B parameters, outperforming much larger models from prior work. Concretely, we generate data using a novel taxonomy, combining difficulty control, personas, and verifier diversification, which allows us to cheaply generate large amounts of terminal environments for RL and SFT training. We open-source our terminal dataset, which is over 2.5x larger than previously released terminal-agent datasets. We then train open-weight models using RL with our data, using a simple, outcome-only recipe. We release our data, models, and code as a strong baseline for future open academic work on terminal agents at https://github.com/hamishivi/tmax.