AgentJudgeBench：一個用於評估LLM裁判在代理型工具呼叫上表現的多難度基準

摘要

LLM評判者被廣泛用於評估代理式工具呼叫系統，然而其在結構化、依賴驅動的工作流程上的可靠性仍未受到充分檢視。我們提出 AgentJudgeBench，這是第一個系統性研究在 workflow DAG 上，LLM 作為評判者對代理式工具呼叫之可靠性的基準測試；此任務有別於更廣泛的開放式文本或偏好評估之 LLM-as-a-judge 任務。該基準包含 3,808 個實例，涵蓋六種 DAG 拓樸與三個難度等級，並以五個生成器（3B-70B 開放權重模型與 GPT-5.4）和六個評判者（20B 至前沿規模）在成對的有真值與無真值條件下進行評估。評判者一致性隨任務難度單調下降，在無真值條件下下降速度快 1.5 倍；而在無真值的困難查詢上，六個評判者無論規模大小均收斂至狹窄的 77–82% 區間，顯示出一個主要受任務難度驅動的結構性上限；雖然對較弱生成器而言，該上限的高度部分取決於提示詞，但僅靠模型容量無法突破。暴露真值並非一律有益：它使 GPT-5.4 的一致性下降 1.5 個百分點，使 Gemini-2.5-Pro 下降 3.9 個百分點，與過度錨定現象一致。在緩解策略中，思維鏈推理與評判者溫度皆影響可忽略，而結構化評估準則最多可提升 6.5 個百分點的一致性，但並非在所有的評判者-生成器配對上均勻適用。在有真值時，QwQ-32B 與程式化參考標準最為吻合；人類驗證研究則指出 GPT-OSS-120B 是與人類判斷最一致的評判者；在無真值時，前沿評判者僅在共同上限內略微領先。這些結果揭露了當前 LLM 評判者的根本限制，並為代理式系統中的可靠評估提供了實用指南。

English

LLM judges are widely used to evaluate agentic tool-calling systems, yet their reliability on structured, dependency-driven workflows remains largely unexamined. We present AgentJudgeBench, the first benchmark to systematically study LLM-as-a-judge reliability for agentic tool-calling over workflow DAGs, as distinct from the broader LLM-as-a-judge task of open-ended text or preference evaluation. The benchmark comprises 3,808 instances spanning six DAG topologies and three difficulty tiers, evaluated with five generators (3B-70B open-weight models and GPT-5.4) and six judges (20B to frontier scale) under paired with- and without-ground-truth conditions. Judge alignment degrades monotonically with task difficulty, 1.5x faster without ground truth, and on hard queries without ground truth all six judges converge to a narrow 77-82% band regardless of scale, revealing a structural ceiling driven primarily by task difficulty, though its height is partly prompt-dependent for weaker generators, that model capacity alone cannot overcome. Ground-truth exposure is not uniformly beneficial: it reduces alignment for GPT-5.4 (1.5 pp) and Gemini-2.5-Pro (3.9 pp), consistent with over-anchoring. Among mitigation strategies, chain-of-thought reasoning and judge temperature both have negligible effect, while structured evaluation rubrics improve alignment by up to 6.5 pp but do not generalize uniformly across judge-generator pairs. With ground truth, QwQ-32B best matches the programmatic reference, while a human validation study identifies GPT-OSS-120B as the most human-aligned judge; without it, frontier judges lead only marginally within the shared ceiling. These results expose fundamental limitations of current LLM judges and yield practical guidelines for reliable evaluation in agentic systems.