CausalDS：評測資料科學智能體中的因果推理

摘要

大型語言模型（LLMs）正日益成為整合型的數據科學代理，將抽象推理與進階工具使用相結合。然而，相關的基準測試領域大致上分裂為兩類：一類是缺乏真實數據分析的符號因果推理基準，另一類則是缺乏具備原則性因果數據生成結構的數據分析基準。此外，現有的因果評估資料集通常僅來自既有來源的精心挑選範例，其多樣性來自有限的模板化變異，而非系統性地生成新穎的合成因果結構。我們提出 CausalDS，這是一個用於評估代理型數據科學工作流程中因果推理能力的基準測試。每個基準測試案例（scene）包含一個取樣自結構因果模型（SCM）的場景，並附有生成的觀測數據，以及一個植基於真實領域的合成自然語言故事。我們可選擇性地將基準測試元件的組成，奠基於從真實世界資料集中獲得的經驗分布，從而在完全合成生成的過程中，保留經驗結構，同時降低「因果鸚鵡」的風險。從每個場景中，我們推導出涵蓋 Pearl 所有三個階層的任務，其中典型的數據科學預測任務歸屬於第一階層。多數任務包含數據科學編碼環節，模型通常需要使用多種工具才能得出最終答案，這是因為任務中經常存在由觀測模型生成的不完美觀測。此外，辨識出問題何時無法得出有根據的答案並予以棄權，會被視為一項有明確評分的結果。因此，該基準測試同時評估了符號因果推理、數據科學、不確定性量化、棄權判斷以及工具使用/編碼能力。

English

Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.