洞察與修復瑕疵：透過代理式資料合成讓視覺語言模型與擴散模型理解視覺偽影

摘要

儘管擴散模型近期有所進展，AI生成影像仍常出現損害真實性的視覺偽影。雖然更全面的預訓練與更大的模型或能減少偽影，但無法保證能完全消除，這使得偽影緩解成為極關鍵的研究領域。過往的偽影感知方法依賴人工標註的偽影資料集，其成本高昂且難以擴展，凸顯了需要自動化方法來可靠獲取偽影標註資料集的需求。本文提出ArtiAgent，能高效創建真實影像與注入偽影的影像配對。該系統包含三個代理：感知代理負責從真實影像中識別並定位實體與子實體，合成代理透過在擴散轉換器中實施新穎的區塊嵌入操作，使用偽影注入工具引入偽影，以及策展代理負責篩選合成後的偽影並為每個實例生成局部與全局解釋。利用ArtiAgent，我們合成了10萬張具有豐富偽影標註的影像，並在多樣化應用中展現其效能與通用性。程式碼公開於連結。

English

Despite recent advances in diffusion models, AI generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, which makes artifact mitigation a highly crucial area of study. Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale, underscoring the need for an automated approach to reliably acquire artifact-annotated datasets. In this paper, we propose ArtiAgent, which efficiently creates pairs of real and artifact-injected images. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images, a synthesis agent that introduces artifacts via artifact injection tools through novel patch-wise embedding manipulation within a diffusion transformer, and a curation agent that filters the synthesized artifacts and generates both local and global explanations for each instance. Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications. Code is available at link.

洞察與修復瑕疵：透過代理式資料合成讓視覺語言模型與擴散模型理解視覺偽影

See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis

摘要

Support