互動式評估需要設計科學

摘要

AI評估正經歷結構性轉變。大型語言模型（LLMs）日益部署為可透過工具、環境、使用者及其他智能體，隨時間推移而運作的系統，然而許多評估實務仍沿用來自以回應為中心的基準測試之假設（例如：固定輸入、孤立輸出，以及可從單一回應作出的結果判斷）。學界已開始建構互動式基準測試，但由此形成的領域圖像零散破碎：各基準在接納何種互動產物、如何為軌跡評分，以及其結果能支撐何種論點上並不一致。本立場論文主張，應將互動式評估視為一項具原則性的評估典範，而非僅是新型智能體基準測試的集合。單純沿用既有評估典範並不足夠。我們將評估定義為從證據到判斷的自動化映射，並指出互動式評估改變了此映射的兩端：證據變成由互動產生的軌跡，而評估程序則必須評判過程、可恢復性、協調性、強健性及系統層級表現。奠基於此定義，我們提出雙軸分類法、推導設計原則與報告標準、檢視代表性場景，並分析長期的評估挑戰如何在軌跡層級重新浮現。

English

AI evaluation is undergoing a structural change. Large language models (LLMs) are increasingly deployed as systems that act over time through tools, environments, users, and other agents, while many evaluation practices still inherit assumptions from response-centered benchmarks (e.g., fixed inputs, isolated outputs, and outcome judgments that can be made from a single response). The field has begun to build interactive benchmarks, but the resulting landscape is fragmented: benchmarks differ in what interaction artifacts they admit, how trajectories are scored, and what claims their results support. This position paper argues that interactive evaluation should be treated as a principled evaluation paradigm, not merely a new family of agent benchmarks. Simply adopting previous evaluation paradigms does not suffice. We define evaluation as an autonomous mapping from evidence to judgments, and show that interactive evaluation changes both sides of this mapping: the evidence becomes interaction-generated trajectories, while the evaluation procedure must assess process, recoverability, coordination, robustness, and system-level performance. Building on this definition, we propose a two-axis taxonomy, derive design principles and reporting standards, examine representative scenarios, and analyze how longstanding evaluation challenges reappear at the trajectory level.