CODA-BENCH: コードエージェントはデータ集約型タスクを処理できるか？

要旨

高级代理正在展现出作为自主工程师运作的潜力，这促使了对能够捕捉现实开发复杂性的评估基准的需求日益增长。此类环境通常同时涉及复杂代码和大规模数据（如文件系统）。然而，现有基准通常孤立地评估以代码为中心或以数据为中心的能力，与实际开发场景存在明显差距。本文通过引入CODA-BENCH来填补这一空白，这是首个在数据密集型环境中联合评估代码智能与数据智能的基准。我们基于Kaggle生态系统构建了一个数据密集型Linux沙箱（包含数百个数据集），代理必须主动探索复杂的文件层次结构以识别相关资源，并为数据驱动型分析任务生成代码。CODA-BENCH包含跨越31个社区的1,009个任务，每个任务环境平均包含980个文件，模拟了真实的数据规模与噪声。对先进代理的评估显示，即使性能最优的系统也难以有效整合数据发现与代码执行，成功率仅为61.1%。这些结果凸显了当前代理在数据密集型任务方面的能力存在显著差距，并为未来研究指明了有前景的方向。

English

Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.