DailyReport：一個評估搜索代理在日常搜索任務上的開放式評測基準

摘要

搜索代理（SAs）通常利用大型語言模型（LLMs）來支援複雜的資訊檢索任務，透過自主探索網路資源並整合資訊，產出全面的回應。在評估搜尋代理時，過往的基準測試主要聚焦於專門任務，這些任務在真實使用者場景中較少出現。此外，這些評測依賴粗略的任務層級評分標準，往往限制了評估的可解釋性。為填補此缺口，我們提出 DailyReport，這是一個開放式基準測試，用於評估搜尋代理在日常搜尋任務上的能力。該基準包含150項開放式任務，搭配3,546條相關評分標準，涵蓋真實使用者廣泛討論且具時效性的資訊需求。每項任務被分解為子任務，並透過階層式評分標準在獨立的維度上進行評估。透過階層式效能歸因與以使用者為中心的彙整方法，我們為每個維度導出高度可解釋的分數，以及一個使用者偏好分數。我們在17個代理系統上的結果顯示，現有系統仍未達到使用者期望。為促進未來研究，我們已將資料集與程式碼公開於 https://github.com/AGI-Eval-Official/DailyReport。

English

Search Agents (SAs) typically leverage large language models (LLMs) to support complex information-seeking tasks by autonomously exploring web sources and synthesizing information into comprehensive responses. For SAs evaluation, prior benchmarks mainly focus on specialized tasks that are unlikely to arise in real-world user scenarios. Moreover, their reliance on coarse task-level rubrics often limits evaluation interpretability. To bridge this gap, we introduce DailyReport, an open-ended benchmark to evaluate SA capabilities on daily search tasks. It contains 150 open-ended tasks with 3,546 associated rubrics, capturing widely discussed and timely information demands of real-world users. Each task is decomposed into subtasks and evaluated with cascade rubrics across disentangled dimensions. Through cascade performance attribution and user-centric aggregation, we derive highly interpretable scores for each dimension, along with a user preference score. Our results on 17 agentic systems show that current systems still fall short of users' expectations. To facilitate future research, our dataset and code are made publicly available at https://github.com/AGI-Eval-Official/DailyReport.