RealSWE：真实用户请求下代码智能体的组合式评估

摘要

编码代理目前通常使用SWE-bench基准系列进行评估，该系列任务基于精选的GitHub问题构建：这些问题篇幅较长、结构清晰且信息丰富。然而，真实用户请求通常要简短得多，且结构化程度较低。为刻画这一差距，我们定义了一个六类信息分类法以及语言风格的四个维度，并将其应用于SWE-chat的真实用户提示以及SWE-bench Verified和Pro中的问题陈述。我们发现，仅包含问题描述（或仅附带有限额外上下文）的请求占真实提示的88%，但在基准问题中仅占7%。此外，87%的真实提示为随意书写风格，而94%的基准问题为正式风格。在这些观察的指导下，我们推出了RealSWE，一个包含381个多变体任务族的数据集，源自SWE-bench Verified和Pro。每个任务族内的变体共享相同的底层任务和参考补丁，仅在信息构成和语言风格上有所不同。使用RealSWE评估七个当代大语言模型，我们发现：i) 真实化输入平均使解决率降低6.4个百分点，且可能改变模型排名。受控分析进一步表明：ii) 包含“期望行为”和“动机”对性能有显著影响，而“环境信息”和“复现步骤”仅增加额外标记却无可测收益；iii) 语言风格的影响较小且因模型而异。这些发现为用户和代理提供了可操作的建议：明确说明期望行为和动机——这是大多数真实提示所省略的——能够显著提升大语言模型的软件工程性能。

English

Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.