SWE-INTERACT：将SWE基准重新构想为用户驱动的长期编码会话

摘要

我们介绍SWE-Interact，这是一个新的测试平台，用于评估编码智能体在多轮、交互式、用户驱动的软件工程任务中的表现。现有前沿的SWE基准测试通常预先提供完整需求，并评估智能体自主实现的能力。相比之下，SWE-Interact将智能体置于真实的开发者工作流程中：一个精心设计的用户模拟器从模糊或不完整的指令开始，逐步揭示需求，检查智能体的工作空间，并提供有针对性的反馈、修改意见和新约束，直到完整的任务目标被传达。基于对真实编码智能体交互的大规模研究，这一设置测试智能体能否发现用户意图、适应不断变化的需求，并基于自身先前的工作进行构建。在一系列前沿和开源模型中，我们发现，在单轮SWE任务中表现强劲并不能可靠地迁移到多轮、用户驱动的工作流程中：表现最佳的模型解决了约50%的单轮基线任务，但仅解决了约25%的相应SWE-Interact任务。我们评估中最强的模型，包括Opus 4.8和GPT 5.5，即使在面对模糊的初始指令时也能强势开局，坚持到用户提出所有需求，更好地整合这些需求并编写干净的代码。然而，它们仍然存在过度自主编码、忘记需求和技术错误的问题。较弱的模型在模糊性下开局不佳，过早放弃，忘记或忽略指令，并更频繁地重写代码。总体而言，SWE-Interact为前沿模型开发衡量了一个正交的现实世界能力轴：交互式目标发现和用户参与的迭代优化。

English

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.