CanvasAgent：通过视觉工具编排实现复杂图像创建与编辑

摘要

复杂的图像创建与编辑任务往往需要超越单一生成或编辑模型的能力。用户需求可能涉及图像合成、目标定位、区域分割、选定内容编辑、中间素材合成、文本读取以及最终结果增强等步骤。这类任务将多模态智能体从基于感知的推理转向以操作为核心的视觉创作——在此过程中，工具必须主动改变视觉状态，而非仅仅进行观察。然而，现有的多模态工具使用智能体大多针对感知、搜索或特定领域编辑进行优化，缺乏用于可执行图像创建轨迹的大规模监督。本文介绍了 CanvasCraft——一个面向复杂图像创建与编辑的大规模多模态工具使用数据集，以及 CanvasAgent——一种工具增强型多模态智能体，能够通过多轮交互学习编排异构视觉工具。CanvasCraft 包含 14 万条完全标注的可执行轨迹和 1 万条强化学习任务规约。CanvasAgent 首先通过监督微训练学习可执行的推理-动作轨迹，随后采用结合结果级与过程级信号的混合奖励，基于 GRPO 进行优化。在运行过程中，CanvasAgent 会检查中间结果、追踪视觉素材，并根据不断演变的视觉状态调整工具决策。实验从最终图像质量与轨迹行为两个维度进行评估，证明了 CanvasAgent 及所提出数据集在复杂多工具图像创建工作流中的有效性。

English

Complex image creation and editing often require more than a single generation or editing model. A user request may involve synthesizing images, localizing objects, segmenting regions, editing selected content, compositing intermediate assets, reading text, and enhancing the final result. Such tasks shift multimodal agents from perception-augmented reasoning to manipulation-centered visual creation, where tools must actively transform visual states rather than merely inspect them. However, existing multimodal tool-use agents are mostly optimized for perception, search, or domain-specific editing, and lack large-scale supervision for executable image-creation trajectories. In this paper, we introduce CanvasCraft, a large-scale multimodal tool-use dataset for complex image creation and editing, and CanvasAgent, a tool-augmented multimodal agent that learns to orchestrate heterogeneous visual tools through multi-turn interaction. CanvasCraft contains 140K fully annotated executable trajectories and 10K RL task specifications. CanvasAgent is first trained with SFT to learn executable reasoning-action trajectories, and is then optimized with GRPO using a hybrid reward that combines outcome- and process-level signals. During rollout, CanvasAgent inspects intermediate results, tracks visual assets, and adapts tool decisions to the evolving visual state. Experiments evaluate both final image quality and trajectory behavior, demonstrating the effectiveness of CanvasAgent and the proposed dataset for complex multi-tool image creation workflows.