IFAdapter：基于实例特征控制的文本到图像生成

IFAdapter: Instance Feature Control for Grounded Text-to-Image Generation

September 12, 2024

作者: Yinwei Wu, Xianpan Zhou, Bing Ma, Xuefeng Su, Kai Ma, Xinchao Wang

cs.AI

摘要

虽然文本到图像（T2I）扩散模型擅长生成单个实例的视觉吸引力图像，但在准确定位和控制多个实例的特征生成方面存在困难。布局到图像（L2I）任务旨在通过将边界框作为空间控制信号来解决定位挑战，但在生成精确实例特征方面仍有不足。为此，我们提出了实例特征生成（IFG）任务，旨在确保生成实例时的位置准确性和特征保真度。为解决IFG任务，我们引入了实例特征适配器（IFAdapter）。IFAdapter通过整合额外的外观标记并利用实例语义地图来对齐实例级特征与空间位置，增强了特征描述。IFAdapter作为即插即用模块引导了扩散过程，使其适用于各种社区模型。为了评估，我们贡献了一个IFG基准，并开发了一个验证流程，客观比较模型生成具有准确位置和特征的实例的能力。实验结果表明，IFAdapter在定量和定性评估中均优于其他模型。

English

While Text-to-Image (T2I) diffusion models excel at generating visually appealing images of individual instances, they struggle to accurately position and control the features generation of multiple instances. The Layout-to-Image (L2I) task was introduced to address the positioning challenges by incorporating bounding boxes as spatial control signals, but it still falls short in generating precise instance features. In response, we propose the Instance Feature Generation (IFG) task, which aims to ensure both positional accuracy and feature fidelity in generated instances. To address the IFG task, we introduce the Instance Feature Adapter (IFAdapter). The IFAdapter enhances feature depiction by incorporating additional appearance tokens and utilizing an Instance Semantic Map to align instance-level features with spatial locations. The IFAdapter guides the diffusion process as a plug-and-play module, making it adaptable to various community models. For evaluation, we contribute an IFG benchmark and develop a verification pipeline to objectively compare models' abilities to generate instances with accurate positioning and features. Experimental results demonstrate that IFAdapter outperforms other models in both quantitative and qualitative evaluations.