ToolArtist：使用工具的统一多模态模型用于智能体图像生成

摘要

文本到图像（T2I）模型能够生成视觉上吸引人的图像，但在需要复杂语义理解、多步推理和外部世界知识整合的开放世界任务上仍然受限。现有工作将智能体能力引入图像生成，但它们要么预设固定工作流，要么仅将开放世界图像生成过程的一部分置于智能体控制之下。因此，推理、工具调用和图像生成并未由单一策略协调。我们提出ToolArtist，一种完全智能体化的图像生成模型，通过对统一多模态模型（UMM）进行后训练获得。ToolArtist在单一统一策略内动态编排推理、外部工具使用和原生图像生成。在监督微调（SFT）阶段，我们为教师智能体配备搜索工具以及图像生成工具。然后，我们将收集到的轨迹转换为UMM兼容格式，其中图像生成工具被隐藏，而生成的图像被保留。在强化学习（RL）阶段，我们为UMM开发了智能体化强化学习基础设施，并引入Reason-Act-Draw GRPO（RAD-GRPO），利用互补的意图奖励和质量奖励联合优化模型。实验表明，将整个开放世界图像生成过程置于智能体策略之下，始终优于采用固定流水线或仅部分组件由智能体控制的方法。我们发布了训练数据和完整的后训练基础设施。

English

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.