PlanBench-XL：評估大規模工具生態系統中LLM工具使用代理的長期規劃

摘要

LLM 代理（LLM agent）日益在大型工具生態系統中運作，現實世界的任務需要發現相關工具、推斷隱含的子目標，並在長時間跨度內適應動態環境。然而，現有的基準測試很少評估在檢索受限的工具可見性下的規劃能力。為了解決這一差距，我們引入了 PlanBench-XL，這是一個包含 327 個零售任務、1,665 個工具的互動式基準測試，旨在測試代理是否能反覆檢索可用的工具，並透過調用這些工具來揭示中間證據，以便後續調用工具最終達成目標。PlanBench-XL 還提供了可選的阻斷機制，透過缺失、失敗或干擾的工具功能來模擬現實世界中的不可預測性，迫使代理在運行時檢測中斷的路徑並進行適應。在十個頂尖 LLM 上的實驗表明，大規模工具規劃仍然具有挑戰性：雖然 GPT-5.4 在無阻斷環境下達到了 51.90% 的準確率，但在最嚴重的阻斷條件下，其準確率驟降至 11.36%。進一步分析顯示，當失敗缺乏明確的錯誤訊號，或恢復需要更長的替代工具使用路徑時，代理尤其脆弱。這些結果將 PlanBench-XL 確立為診斷代理規劃失敗的測試平台，並凸顯了在具有大規模、不完美工具環境的長期任務中，需要強健的自適應規劃能力。

English

LLM agents increasingly operate in large tool ecosystems, where real-world tasks require discovering relevant tools, inferring implicit sub-goals, and adapting to dynamic environments over long horizons. However, existing benchmarks rarely evaluate planning under retrieval-limited tool visibility. To address this gap, we introduce PlanBench-XL, an interactive benchmark of 327 retail tasks over 1,665 tools that tests whether agents can iteratively retrieve usable tools, invoke them to uncover intermediate evidence for subsequent calls toward the final goal. PlanBench-XL further features an optional blocking mechanism that simulates real-world unpredictability through missing, failing, or distracting tool functions, forcing agents to detect disrupted paths and adapt at runtime. Experiments on ten leading LLMs show that massive-tool planning remains challenging: while GPT-5.4 achieves 51.90% accuracy in block-free settings, it collapses to 11.36% under the most severe blocking condition. Further analysis shows that agents are especially vulnerable when failures lack explicit error signals or when recovery requires longer alternative tool-use paths. These results establish PlanBench-XL as a testbed for diagnosing agentic planning failures and highlight the need for robust adaptive planning in long-horizon tasks with large, imperfect tool environments.