SWE-Together：在交互式用户会话中评估编码代理

摘要

大多数编码智能体基准测试是静态的：智能体预先接收完整的任务描述，并且仅根据其最终代码进行评判。而真实的编码辅助是交互式的，用户会在多轮对话中明确目标、添加约束并纠正错误。我们提出了 SWE-Together，这是一个从真实用户与智能体编码会话中重构的多轮基准测试。为了使真实交互可验证，我们从11,260个记录的会话中精选出109个仓库级任务，选取那些具有可恢复仓库状态、清晰用户目标和可观测结果的会话。为了在不同的智能体间重演这些交互，我们构建了一个基于反应式大语言模型的用户模拟器，该模拟器保留了原始用户的意图，并在编码智能体进展需要时提供反馈。为了评估作为协作者的智能体，我们同时衡量最终仓库的正确性以及交互过程中所需的纠正性反馈轮次数量。与前沿编码智能体的实验表明，更强的智能体通常能达到更高的最终成功率，同时需要的干预次数更少，这表明用户体验得到了改善。

English

Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users' intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.