DSAgentBench：智能体能否在真实计算机环境中自动化端到端数据科学工作流？

摘要

真实世界的数据科学涉及长周期工作流，涵盖数据整理、探索、建模、可视化和验证，并且需要在真实运行环境中协调使用 Notebook、IDE、终端、浏览器和数据库等工具。然而，现有基准缺乏真实计算机交互，无法评估智能体能否在真实计算环境中执行完整的端到端数据科学工作流，因而未能体现数据科学实践的多阶段、多工具特点。我们提出 DSAgentBench，这是首个在真实计算机环境中评估智能体能否自动化完整数据科学工作流的基准。DSAgentBench 包含 275 个多样化任务，覆盖数据科学全生命周期，反映了实际场景中所需的复杂性和工具协调能力。每个任务都要求智能体依据中间输出进行决策并协调使用工具，同时配备一个确定性评估器，用于验证分析正确性、可视化输出和模型性能，而非仅检查代码执行。我们对 15 个闭源和开源模型进行了大量实验，结果表明，即使是最强的智能体 Claude-4.6-Sonnet，也仅达到 56.70% 的任务成功率，而所有开源智能体的成功率均低于 1%，并且经常在工具编排、操作系统环境关联和多步推理上失败。这些结果揭示了当前智能体系统与真实数据科学工作流之间的巨大能力差距，使 DSAgentBench 成为开发基于真实环境、可验证、自主的数据科学智能体的基础。我们已在 https://github.com/vis-nlp/DSAgentBench 上发布 DSAgentBench。

English

Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.