多智能体自动化研究系统词汇集

摘要

我们提出了一套用于自动化研究系统的术语表，这些系统由一个或多个智能体构建，旨在使设计选择更易于描述和比较。该术语表明确了：1) 智能体是谁；2) 系统中可用哪些操作；3) 谁可以调用这些操作；4) 智能体如何通信；5) 单次运行及多次运行间哪些信息可见；6) 如何选择下一步行动；7) 如何启动一次运行；8) 如何评估输出结果。轨迹记录的是从输入任务到最终产出物的完整运行过程。由于智能体、操作和初始化过程可能具有随机性，同一任务上的多次运行会在轨迹上形成分布，而非单一行为模式。我们的术语表将结构性设计问题（例如智能体何时通信、何时获得或失去某项能力、何时跨运行传递信息）转化为可检验的选项。同时，它将评估器纳入系统组件，因为报告的性能提升取决于代理评分与真实质量之间的匹配程度。这种拆分也将“这些系统缺乏品味”这一模糊批评分解为两种具有不同解决方案的失效模式：生成品味指系统在观察到任何评分之前提出新颖轨迹的速率；评估品味则指代理评分与应匹配的真实质量之间的差距。我们以近期自动研究系统为例实例化该术语表，以证明其能够覆盖结构差异显著的设计。

English

We introduce a vocabulary for automated research systems built from one or more agents to make their design choices easier to describe and compare. The vocabulary specifies 1) who the agents are, 2) what operations are available in the system, 3) who may invoke them, 4) how agents communicate, 5) what information is visible within and across runs, 6) how the next action is chosen, 7) how a run begins, and 8) how outputs are evaluated. A trajectory records one run from the input task to the returned artifact. Because agents, operations, and initialization may be stochastic, repeated runs on the same task induce a distribution over trajectories rather than a single behavior. Our vocabulary turns structural design questions, such as when agents should communicate, gain or lose a capability, or carry information across runs, into testable choices. It also makes the evaluator a component of the system, since reported gains depend on how closely the proxy score matches true quality. That separation also splits the vague complaint that these systems lack taste into two failures with different solutions. Generative taste is the rate at which a system proposes novel trajectories before any score is observed, and evaluative taste is the gap between the proxy score and the quality it should match. We instantiate the vocabulary on recent autoresearch systems to illustrate that it covers designs that differ widely in structure.