多智能體自動化研究系統的詞彙表

摘要

我們引入一個由一個或多個智能體構建的自動化研究系統的詞彙表，以便更輕鬆地描述和比較其設計選擇。該詞彙表明確說明了：1) 智能體的身份，2) 系統中可用的操作，3) 誰可以調用這些操作，4) 智能體之間如何通信，5) 單次運行及跨運行中可見的資訊，6) 下一步行動如何選擇，7) 運行如何開始，以及8) 輸出如何評估。一條軌跡記錄了從輸入任務到返回產物的完整運行過程。由於智能體、操作和初始化可能具有隨機性，對相同任務進行多次運行會產生軌跡分佈，而非單一的行為。我們的詞彙表將結構性設計問題（例如智能體何時應進行通信、獲取或失去某項能力、或在運行之間傳遞資訊）轉化為可測試的選擇。該詞彙表還將評估器視為系統的一個組成部分，因為報告的性能提升取決於代理評分與真實品質的接近程度。這種分離也將「這些系統缺乏品味」這一模糊批評拆解為兩個具有不同解決方案的失敗：生成品味是指系統在觀察到任何評分之前提出新穎軌跡的速率，而評估品味則是代理評分與其應匹配的品質之間的差距。我們以近期多個自動研究系統為例來實例化該詞彙表，以說明它能夠涵蓋結構差異巨大的設計。

English

We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare. The vocabulary specifies 1) who the agents are, 2) what operations are available in the system, 3) who may invoke them, 4) how agents communicate, 5) what information is visible within and across runs, 6) how the next action is chosen, 7) how a run begins, and 8) how outputs are evaluated. A trajectory records one run from the input task to the returned artifact. Because agents, operations, and initialization may be stochastic, repeated runs on the same task induce a distribution over trajectories rather than a single behavior. Our vocabulary turns structural design questions, such as when agents should communicate, gain or lose a capability, or carry information across runs, into testable choices. It also makes the evaluator a component of the system, since reported gains depend on how closely the proxy score matches true quality. That separation also splits the vague complaint that these systems lack taste into two failures with different solutions. Generative taste is the rate at which a system proposes novel trajectories before any score is observed, and evaluative taste is the gap between the proxy score and the quality it should match. We instantiate the vocabulary on recent autoresearch systems to illustrate that it covers designs that differ widely in structure.