SWE-INTERACT：重新構想SWE基準為用戶驅動的長週期編程會話

摘要

我們介紹SWE-Interact，一個新的測試平台，用於評估程式編碼代理在多輪、互動式、使用者驅動的軟體工程任務中的表現。現有的前沿SWE基準測試通常會事先提供完整的需求，並評估代理自主完成任務的能力。相比之下，SWE-Interact將代理置於真實的開發者工作流程中：一個精心設計的使用者模擬器從模糊或不完整的指令開始，逐步揭示需求，檢查代理的工作空間，並提供針對性的回饋、修訂與新限制，直到所有任務目標被完整傳遞為止。此設置基於對真實編碼代理互動的大規模研究，測試代理是否能發現使用者意圖、適應不斷變化的需求，並在其先前工作的基礎上持續建構。在一系列前沿與開源模型測試中，我們發現，在單輪SWE任務中表現優異的模型，並不一定能可靠地遷移至多輪、使用者驅動的工作流程：表現最佳的模型能解決約50%的單輪基準任務，但在對應的SWE-Interact任務中僅能解決約25%。在我們評估中最強大的模型，包括Opus 4.8與GPT 5.5，即使在面對模糊的初始指令時也能有良好的開端，堅持到使用者提出所有需求，更好地整合需求並撰寫乾淨的程式碼。然而，它們仍存在過度主動編碼、遺忘需求以及技術錯誤等問題。較弱的模型在不確定性下表現不佳，過早放棄、忘記或忽略指令，並進行更多的程式碼重構。總體而言，SWE-Interact為前沿模型發展衡量了一個正交的現實世界能力軸：與使用者互動中的目標發現與迭代優化。

English

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.