DSAgentBench：智能體能否在真實電腦環境中自動化端到端資料科學工作流程？

摘要

真實世界的資料科學涉及涵蓋資料整理、探索、建模、視覺化與驗證的長期工作流程，並要求在真實作業環境中協調使用筆記本、整合開發環境（IDE）、終端機、瀏覽器與資料庫等工具。然而，現有的評測基準缺乏真實電腦互動，且未評估代理能否在真實運算環境中執行完整的端對端資料科學工作流程，因而未能捕捉資料科學實務中多階段、多工具的特性。我們提出 DSAgentBench，這是首個評估代理能否在真實電腦環境中自動化完整資料科學工作流程的評測基準。DSAgentBench 包含 275 項多元任務，涵蓋完整的資料科學生命週期，反映實務所需的複雜度與工具協調能力。每項任務都要求將決策建立在中間輸出之上並協調使用工具，同時配備確定性評估器，用以驗證分析正確性、視覺化輸出與模型效能，而非僅檢查程式碼執行。我們對 15 個閉源與開源模型進行的廣泛實驗顯示，即使是最強大的代理 Claude-4.6-Sonnet，任務成功率僅達 56.70%，而所有開源代理的成功率皆低於 1%，且經常在工具編排、作業系統落地與多步驟推理上失敗。這些結果揭示了當前代理系統與真實資料科學工作流程之間存在顯著的能力差距，使 DSAgentBench 成為發展具落地性、可驗證且自主運作的資料科學代理的基礎。我們於 https://github.com/vis-nlp/DSAgentBench 發布 DSAgentBench。

English

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.