IFAdapter：基於實例特徵控制的文本對圖像生成

IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

September 12, 2024

作者: Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, Xinchao Wang

cs.AI

摘要

儘管文字轉圖像（T2I）擴散模型擅長生成個別實例的視覺吸引力圖像，但在準確定位和控制多個實例的特徵生成方面卻遇到困難。布局轉圖像（L2I）任務旨在解決定位挑戰，透過將邊界框納入空間控制信號，但在生成精確實例特徵方面仍有不足。為此，我們提出了實例特徵生成（IFG）任務，旨在確保生成實例時的位置準確性和特徵保真度。為應對IFG任務，我們引入了實例特徵適配器（IFAdapter）。IFAdapter通過納入額外的外觀令牌並利用實例語義地圖來對齊實例級特徵與空間位置，增強特徵描述。IFAdapter作為即插即用模塊引導擴散過程，使其適應各種社區模型。為了評估，我們提出了一個IFG基準測試集，並開發了一個驗證流程，客觀比較模型生成具有準確位置和特徵的實例的能力。實驗結果表明，IFAdapter在定量和定性評估中均優於其他模型。

English

While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.