Counsel：面向代理任務的元評估資料集

摘要

隨著代理系統處理越來越複雜的多步驟任務，評估其軌跡成為一大瓶頸——在流行的代理基準測試中，對單一軌跡進行人工標註可能需要數小時，使得在衡量效能或整理訓練資料時難以大規模評估。這促使業界廣泛依賴自動化方法，例如將LLM作為評判者（LLMJ），以在流程層級與結果層級大規模評析代理系統；然而，LLMJ評析的可靠性往往未經檢驗。為此，我們推出Counsel，這是首個針對代理任務的後設評估公開資料集。Counsel包含來自開放權重LLMJ在兩個代理基準上的流程層級評析：tau-bench（客戶支援代理）與DA-Code（程式碼代理），以及人類對這些評析的後設評估。人工標註者將每項被標記錯誤的評析歸類為「一針見血」、「位置正確但推論不佳」或「不應標記」，並達到可靠的標註者間一致性（克里本多夫α係數為0.78）。以此產生的資料集根據人類一致性，將LLMJ的評析按錯誤在軌跡中的位置與推論品質分層，成為校準、改進或訓練代理專用LLMJ的寶貴資料。比較開放權重評判模型後，我們發現更強大的評判模型與更多推論投入都能提升與人類的一致性，其中最強的評判模型在位置辨識上達到約88%的一致性，在推論品質上則約為65%。Counsel使用開放權重模型生成，並採用寬鬆授權供社群廣泛使用，我們希望這能促進對基於LLM的代理系統評估者進行嚴謹研究，並進一步提升其與人類的對齊。

English

As agentic systems tackle increasingly complex multi-step tasks, evaluating their trajectories presents a major bottleneck - human annotation of a single trajectory on popular agentic benchmarks can take hours, making it difficult to scale evaluations for measuring performance or curating training data. This has driven widespread reliance on automated approaches such as LLM-as-a-judge (LLMJ) to critique agents at the process and outcome-levels at scale, however, the soundness of LLMJ critiques often goes unmeasured. Here, we introduce Counsel, the first public dataset of meta-evaluations for agentic tasks. Counsel consists of process-level critiques from open-weight LLMJs on two agent benchmarks: tau-bench (customer support agents) and DA-Code (coding agents), and human meta-evaluations of these critiques. Human annotators label critiques on each flagged error as "spot on", "correct location but poor reasoning", or "should not have flagged", achieving reliable inter-annotator agreement (Krippendorff's alpha of 0.78). The resulting dataset stratifies LLMJ critiques by human alignment across both error location within a trajectory and reasoning quality, serving as valuable data to calibrate, improve, or train LLMJs for agents. Comparing open-weight judges, we find that more capable judge models and more reasoning effort both enabled improved human agreement, with the strongest judge reaching ~88% agreement on location and ~65% on reasoning. Counsel is generated using open-weight models and is permissively licensed for broad community use, which we hope will enable rigorous study and improved alignment of LLM-based evaluators for agentic systems.