SWE-Together：在互動式使用者會話中評估編碼代理

摘要

大多數編碼代理基準測試均為靜態：代理會事先收到完整的任務描述，並僅根據其最終代碼進行評判。然而，實際的編碼輔助是互動式的，使用者會在多輪對話中釐清目標、新增限制條件並修正錯誤。為此，我們提出了SWE-Together，這是一個從真實使用者與代理互動會話中重建的多輪基準測試。為了使真實互動具備可驗證性，我們從11,260個記錄會話中篩選出109個倉庫級任務，選取那些倉庫狀態可復原、使用者目標明確且結果可觀察的會話。為了在不同代理間重現這些互動，我們建構了一個反應式的基於大型語言模型的使用者模擬器，該模擬器保留原始使用者的意圖，並在編碼代理的進展需要時提供反饋。為了評估代理作為協作者的能力，我們同時衡量最終倉庫的正確性，以及互動過程中所需的糾錯反饋輪數。對前沿編碼代理的實驗表明，較強的代理通常能實現更高的最終成功率，同時所需的人工干預次數更少，這意味著使用者體驗得到改善。

English

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.