代理式視覺生成：從生成式模型到代理式控制

摘要

視覺生成正從透過單次呼叫使用的生成模型，演變為能規劃、選擇工具、檢查中間合成輸出、修正失敗並重用先前經驗的代理式控制流程。在大多數現有系統中，控制器是 LLM 或 VLM，而視覺生成模型則作為工具或執行器。然而，現有研究缺乏一致的判準來判斷生成系統何時成為代理式。規劃深度、工具使用、多角色協作與強化學習常被視為代理性的證據，儘管這些都不必然決定控制器能做出哪些生成決策。我們依據控制器在生成過程中能直接控制什麼來組織這個領域。在 L1 條件控制中，控制器為預先確定的生成器準備輸入，但不控制執行哪一種視覺操作。在 L2 執行控制中，它選擇並呼叫實際的生成、編輯、渲染或其他修改內容的操作。在 L3 結果自適應控制中，它觀察中間結果，並利用該觀察改變當前任務中的後續操作。在 L4 經驗自適應控制中，它保留已完成任務的經驗，並利用該經驗改變未來任務的決策。L0 固定支援另行指稱未部署能做出生成層級決策之控制器的生成器、編輯器、評估器、獎勵模型、基準測試與固定管線。這些層級描述的是逐步擴大的決策範圍，而非模型大小、系統複雜度、輸出品質、工具或角色數量，或訓練方法。將此框架應用於影像、影片、編輯、3D、世界、投影片與使用者介面生成，可揭示控制器能力如何演變，以及其機制如何分佈於各層級。

English

Visual generation is evolving from generative models used through a single invocation into agentic control processes that can plan, select tools, inspect intermediate synthesized outputs, revise failures, and reuse prior experience. In most existing systems, the controller is an LLM or VLM, while visual generation models serve as tools or executors. However, existing work lacks a consistent criterion for determining when a generation system becomes agentic. Planning depth, tool use, multi-role collaboration, and reinforcement learning are often treated as evidence of agenticity, even though none of them necessarily determines which generation decisions the controller can make. We organize the field according to what the controller can directly control in the generation process. At L1 Conditioning Control, the controller prepares the input to a predetermined generator but does not control which visual operation is executed. At L2 Execution Control, it selects and invokes actual generation, editing, rendering, or other content-modifying operations. At L3 Outcome-Adaptive Control, it observes an intermediate outcome and uses that observation to change a subsequent operation within the current task. At L4 Experience-Adaptive Control, it retains experience from completed tasks and uses that experience to change decisions on future tasks. L0 Fixed Support separately denotes generators, editors, evaluators, reward models, benchmarks, and fixed pipelines without a deployed controller that makes generation-level decisions. These levels describe a progressively broader decision-making scope rather than model size, system complexity, output quality, tool or role count, or training method. Applying this framework across image, video, editing, 3D, world, slide, and user-interface generation reveals how controller capabilities have evolved and how their mechanisms are distributed across levels.