智能体检索基准：评估编码智能体的仓库上下文检索

摘要

现代编码代理通常通过是否最终生成正确的补丁来评估，但补丁生成依赖于早期的上下文获取阶段：即找到任务所需的仓库文件。我们提出Agent Retrieval Bench，这是一个针对该上游检索问题的文件级基准。样本基于真实编码工作流信号构建，并在冻结的基础提交仓库上进行评估，相关性由代理下一步需要的内容而非直接查询与文件的语义相似度来定义。该基准涵盖四项正检索任务：code2test、comment2context、trace2code和edit2ripple；第五个子集利用基于自然证据的无金标准案例和反事实的错误仓库控制来评估选择性检索。Agent Retrieval Bench包含来自25个仓库的427个样本：345个正例、50个自然无金标准示例和32个反事实控制项。语料库包括308个基础提交快照、392,000个文件和790万个块。我们评估了词汇检索、RepoMap、开源嵌入、选择性弃权以及记录的代理上下文选择。没有单一检索家族占据主导地位：Qwen3-Embedding-4B在正样本上获得最佳的样本加权MRR，Qwen3-Embedding-8B获得最佳的Recall@20，而RepoMap在8K token预算下获得最佳的上下文产出，不同任务层面的胜出者差异显著。使用反事实控制校准的选择性阈值并未提高自然无金标准案例上的选择性成功，揭示了校准差距。记录的轨迹在27%至35%的样本中也遗漏了所有金标准文件。一项受控的种子干预试验发现，与随机非金标准上下文相比，检索得到的初始上下文在种子后探索中能获得更高的文件F1分数，而Oracle金标准上下文则显示出显著的剩余提升空间。

English

Modern coding agents are usually evaluated by whether they eventually produce a correct patch, but patch generation depends on an earlier context-acquisition stage: finding the repository files needed for the task. We introduce Agent Retrieval Bench, a file-level benchmark for this upstream retrieval problem. Samples are built from real coding-workflow signals and evaluated against frozen base-commit repositories, with relevance defined by what an agent needs next rather than direct query-file semantic similarity. The benchmark covers four positive-retrieval tasks: code2test, comment2context, trace2code, and edit2ripple; a fifth subset evaluates selective retrieval using natural evidence-backed no-gold cases and counterfactual wrong-repository controls. Agent Retrieval Bench contains 427 samples across 25 repositories: 345 positive examples, 50 natural no-gold examples, and 32 counterfactual controls. The corpus includes 308 base-commit snapshots, 392,000 files, and 7.9 million chunks. We evaluate lexical retrieval, RepoMap, open-source embeddings, selective abstention, and logged agent context selection. No single retrieval family dominates: Qwen3-Embedding-4B has the best sample-weighted MRR on positive samples, Qwen3-Embedding-8B the best Recall@20, and RepoMap the best budgeted context yield at 8K tokens, with task-level winners differing substantially. Selective thresholds calibrated with counterfactual controls do not improve selective success on natural no-gold cases, revealing a calibration gap. Logged trajectories also miss every gold file on 27-35 percent of samples. A controlled seed-intervention pilot finds that retrieval-derived initial context yields higher file F1 with less post-seed exploration than random non-gold context, while oracle gold context shows substantial remaining headroom.