SWE-Touch：當用戶觸及程式碼時的編程智能體基準測試

摘要

現實世界的軟體開發要求編碼代理在共享工作區中運作，在這種環境中，用戶可能在任務進行中檢查和修改代碼，然而現有的倉儲庫級別基準測試通常評估單獨工作的代理，或將用戶參與限制為訊息形式。這使我們提出疑問：編碼代理如何理解共享工作區中的代碼變更並做出回應？我們引入SWE-Touch，一個通過經過驗證的對抗性編輯（Counter-Edits）來壓力測試這種情境的框架：對任務相關代碼進行的合理編輯，但與任務完成相衝突。SWE-Touch從多個修復軌跡中挖掘任務關鍵區域，使用單獨的用戶補丁生成器來構建編輯，並在代理到達相關代碼時將編輯與情境化用戶訊息一起注入。我們在SWE-bench Verified上評估了九個編碼模型，並在SWE-Bench Pro和DeepSWE的更長時程任務上進行了額外實驗。對抗性編輯使SWE-bench Verified的平均解決率降低了7.7個百分點，退化現象在兩個更長時程的基準測試中持續存在。軌跡分析將這些失敗歸因於對不斷演變的工作區的有限感知：代理可能保留衝突的代碼，或在未充分重新檢查倉儲庫並用針對性測試驗證修訂後代碼的情況下將其替換。這些發現表明，強大的自主性能尚不能確保共享工作區協作所需的狀態感知和適應性行為，並指出檢測工作區變化、協調衝突編輯與任務之間的矛盾、以及驗證受影響行為是未來優化的關鍵能力。

English

Real-world software development requires coding agents to operate in shared workspaces where users may inspect and modify code during an ongoing task, yet existing repository-level benchmarks typically evaluate agents working alone or restrict user participation to messages. This leads us to ask: how do coding agents understand and respond to code changes in a shared workspace? We introduce SWE-Touch, a framework that stress-tests this setting through validated Counter-Edits: plausible edits to task-relevant code that conflict with task completion. SWE-Touch mines task-critical regions from multiple repair trajectories, uses a separate User Patch Generator to construct the edits, and injects them with contextual user messages when agents reach the relevant code. We evaluate nine coding models on SWE-bench Verified, with additional experiments on longer-horizon tasks from SWE-Bench Pro and DeepSWE. Counter-Edit lowers average resolve rate by 7.7 percentage points on SWE-bench Verified, with degradation also persisting on both longer-horizon benchmarks. Trajectory analysis links these failures to limited awareness of the evolving workspace: agents may retain conflicting code or replace it without sufficiently re-inspecting the repository and validating the revised code with targeted tests. These findings show that strong autonomous performance does not yet ensure the state awareness and adaptive behavior needed for shared-workspace collaboration, and point to detecting workspace changes, reconciling conflicting edits with the task, and verifying the affected behavior as key capabilities for future optimization.