「ScrambleToolBench：智能體即使自身地圖已指向下一步，仍會窮舉搜尋」

摘要

要在開放世界環境中穩健運作，自主智能體應能僅透過互動推斷陌生系統的行為，即使在缺乏文件的情況下亦然。然而，現有的工具使用基準在靜態環境中揭露語義工具架構，使智能體得以依賴先備知識而非自主探索。為解決此限制，我們引入 ScrambleToolBench——一個專為隔離行為推理所設計的互動式終端基準。透過移除語義線索並實施連續的任務課程，該基準要求智能體完全經由試誤互動來發掘隱藏的工具行為。此基準進一步引入動態挑戰，包括映射漂移、隨機行動失敗及時間執行窗口，以評估智能體能否在環境變化時修訂並調整其假設。我們對當前最先進語言模型的評測顯示，成功的初始發現並不能轉化為穩健的適應能力。面對映射漂移等結構性變化時，智能體無法運用諸如循環追蹤等演繹策略，反而表現出信念慣性，或退而訴諸窮舉搜尋。增加測試時推理只會放大這種昂貴的暴力搜尋，而非促成演繹性恢復。雖然為智能體配備持久記憶可減少複合錯誤，但它們仍無法有效推斷結構性變化，凸顯了當前智能體推理能力的缺口。

English

To operate robustly in open-world environments, autonomous agents should be able to infer the behavior of unfamiliar systems through interaction alone, even in the absence of documentation. However, existing tool-use benchmarks expose semantic tool schemas in static environments, allowing agents to rely on prior knowledge rather than autonomous discovery. To address this limitation, we introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning. By removing semantic cues and enforcing a continuous task curriculum, the benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction. The benchmark further introduces dynamic challenges, including mapping drift, stochastic action failures, and temporal execution windows, to evaluate whether agents can revise and adapt their hypotheses as the environment changes. Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation. When faced with structural changes such as mapping drift, agents fail to use deductive strategies such as cycle tracing, and instead exhibit belief inertia or fall back to exhaustive search. Increasing test-time reasoning only amplifies this expensive brute-force search rather than enabling deductive recovery. While equipping agents with persistent memory reduces compounding errors, they remain unable to efficiently infer structural changes, highlighting a gap in current agent reasoning.