AgentJudgeBench：用于评估LLM在智能体工具调用中评判能力的高难度多级基准

摘要

LLM评判器被广泛用于评估智能体工具调用系统，然而它们在结构化、依赖驱动工作流上的可靠性在很大程度上仍未得到检验。我们提出AgentJudgeBench，这是首个系统性地研究LLM作为评判器在工作流DAG智能体工具调用任务中可靠性的基准，区别于更广泛的开放式文本或偏好评估的LLM评判器任务。该基准包含3,808个实例，涵盖六种DAG拓扑结构和三个难度层级，使用五种生成器（30亿至700亿参数的开权重模型以及GPT-5.4）和六种评判器（200亿参数至前沿规模）在有无标准答案配对条件下进行评估。评判器对齐度随任务难度单调下降，无标准答案时的下降速度是有标准答案时的1.5倍；在无标准答案的困难查询上，所有六种评判器无论规模大小均收敛于狭窄的77%-82%区间，揭示了主要由任务难度驱动的结构性天花板，尽管其高度对较弱的生成器部分依赖于提示词，但模型容量本身无法突破这一限制。标准答案的暴露并非均匀有利：它降低了GPT-5.4（1.5个百分点）和Gemini-2.5-Pro（3.9个百分点）的对齐度，与过度锚定效应一致。在缓解策略中，思维链推理和评判器温度均影响甚微，而结构化评估量规可将对齐度提升最多6.5个百分点，但无法在评判器-生成器对之间均匀泛化。在有标准答案的条件下，QwQ-32B与程序化参考最匹配，而人工验证研究识别GPT-OSS-120B为最符合人类对齐的评判器；在无标准答案的条件下，前沿评判器仅在共有天花板内略微领先。这些结果揭示了当前LLM评判器的根本局限性，并为智能体系统的可靠评估提供了实用指南。

English

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.