ExecRetrieval：衡量程式碼嵌入檢索中的功能正確性差距

摘要

基於嵌入的程式碼檢索是程式設計代理與檢索增強程式碼生成的核心組成部分，在這些應用中，檢索到正確的程式碼比檢索到詞彙相似的程式碼更為重要。現有的程式碼檢索基準並未在搜尋池中植入受控的、經執行驗證的單次編輯變體，這些變體對應每個查詢的標準實作，因此，在檢索環境中，嵌入是否能從功能上區分正確程式碼與近乎複製但不正確的程式碼，仍是未解的問題。要解決這個問題，需要一個基準，其搜尋池本身就包含相關的反事實變體——也就是與每個標準實作幾乎相同且經執行驗證的錯誤變體——如此一來，檢索器的排序便可直接檢驗其功能區分能力，而非僅是主題或身分重疊。我們提出 ExecRetrieval，包含 939 個 Python 任務，每個任務配有一個經執行驗證的標準實作，以及最多四個經執行驗證的錯誤干擾項；每個干擾項皆由機械式突變生成，僅做單一目標編輯。我們在提供者原生呼叫下，評估了 23 種稠密嵌入配置及 BM25，並使用配對 McNemar 檢定與查詢層級的 bootstrap 信賴區間進行分析。在搜尋池中加入近乎複製的反事實變體後，表現最佳的主機代管系統達到 exec@10 = 1.00，但 exec@1 僅為 0.331；在四個領先系統中，排名第一時的漏失結果有 91.5% 至 99.4% 是配對的錯誤變體，而且在這些領先系統上，有 67% 至 78% 的查詢中，標準實作的分數低於其四個配對干擾項中至少一個。完整的資料集、執行驗證器、嵌入矩陣、環境快照與成對統計檢定結果均已發布於附錄 D 的網址。

English

Embedding-based code retrieval is a core component of coding agents and retrieval-augmented code generation, where retrieving correct code matters more than retrieving lexically similar code. Existing code-retrieval benchmarks do not plant controlled, execution-verified single-edit variants of each query's canonical implementation in the search pool, leaving the question of whether embeddings can functionally discriminate correct from near-clone-but-incorrect code unanswered in a retrieval setting. Resolving this requires a benchmark whose search pool itself contains the relevant counterfactuals -- execution-verified buggy variants near-identical to each canonical -- so that a retriever's rank ordering can be directly tested for functional discrimination rather than topical or identity overlap. We introduce ExecRetrieval, 939 Python tasks each paired with one execution-verified canonical implementation and up to four execution-verified buggy distractors, each generated by a mechanical mutation making a single targeted edit, and evaluate 23 dense embedding configurations plus BM25 under provider-native invocation with paired McNemar tests and query-level bootstrap intervals. With near-clone counterfactuals in the pool, the top hosted system reaches exec@10 = 1.00 but only exec@1 = 0.331; rank-1 misses are paired buggy variants 91.5-99.4% of the time across the four leading systems, and the canonical scores below at least one of its four paired distractors in 67-78% of queries on the leading systems. The full dataset, execution oracle, embedding matrices, environment snapshot, and pairwise statistical tests are released at the URL in Appendix D.