ExecRetrieval：衡量代码嵌入检索中的功能正确性差距

摘要

基于嵌入的代码检索是代码智能体和检索增强代码生成的核心组成部分；在此类任务中，检索到正确的代码比检索到词法相似的代码更重要。现有代码检索基准没有将“受控的、经过执行验证的单次编辑变体”植入搜索池——即对每条查询的标准实现进行一次编辑后得到的变体——因此，嵌入表示能否在功能层面区分正确代码与近似克隆但有缺陷的代码，这在检索场景中仍是一个悬而未决的问题。要解决它，需要一个其搜索池本身就包含相关反事实样本的基准：这些反事实样本是经执行验证的缺陷变体，与每个标准实现几乎完全相同。这样才能直接检验检索器的排序结果是在做功能判别，而非仅仅基于主题或文本同一性重叠。为此，我们提出 ExecRetrieval。它包含 939 个 Python 任务，每个任务对应一个经执行验证的标准实现和至多四个经执行验证的缺陷干扰项；每个干扰项均由机械式变异生成，只做一次有针对性的编辑。我们在提供方原生调用方式下评估了 23 种稠密嵌入配置以及 BM25，并使用配对 McNemar 检验和查询级 bootstrap 置信区间进行分析。当搜索池中存在近克隆反事实样本时，表现最好的托管系统 exec@10 达到 1.00，但 exec@1 仅为 0.331；在四个领先系统上，当首位检索未命中时，91.5%–99.4% 的未命中结果就是配对的缺陷变体；并且在 67%–78% 的查询中，标准实现的得分低于其四个配对干扰项中的至少一个。完整数据集、执行预言机、嵌入矩阵、环境快照以及两两统计检验均已发布，详见附录 D 中的 URL。

English

Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.