ToolArtist：用於代理式圖像生成的工具使用統一多模態模型

摘要

文字轉圖像（T2I）模型能夠產生視覺上引人入勝的圖像，但在需要複雜語意理解、多步推理及整合外部世界知識的開放世界任務上仍有所限制。現有研究嘗試將代理能力引入圖像生成，但這些方法不是預先規定固定工作流程，就是僅將開放世界圖像生成過程中的一部分交由代理控制。因此，推理、工具呼叫與圖像生成並非由單一策略協調。我們提出ToolArtist，一種透過對統一多模態模型（UMM）進行後訓練所獲得之完全代理式圖像生成模型。ToolArtist在單一統一策略中動態編排推理、外部工具使用與原生圖像生成。在監督式微調（SFT）階段，我們為教師代理配備搜尋工具與圖像生成工具。接著，我們將收集到的軌跡轉換為UMM相容格式，其中圖像生成工具被隱藏，而產生的圖像則被保留。在強化學習（RL）階段，我們為UMM開發了代理式RL基礎設施，並引入Reason-Act-Draw GRPO（RAD-GRPO），利用互補的意圖與品質獎勵來聯合優化模型。實驗結果顯示，將整個開放世界圖像生成流程置於代理策略之下，一貫地優於固定管線或僅部分代理控制元件的方法。我們釋出訓練資料與完整的後訓練基礎設施。

English

Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.