InstantStyle：文本到图像生成中的风格保留免费午餐

InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

April 3, 2024

作者: Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen

cs.AI

摘要

无调优扩散模型在图像个性化和定制领域展现了显著潜力。然而，尽管取得了这一显著进展，当前模型在生成风格一致的图像时仍面临若干复杂挑战。首先，风格概念本身具有不确定性，涵盖颜色、材质、氛围、设计、结构等多个方面。其次，基于反演的方法易导致风格退化，常造成细节丢失。最后，基于适配器的方法往往需要对每张参考图像进行精细的权重调优，以在风格强度和文本可控性之间取得平衡。本文首先探讨了几项引人注目但常被忽视的观察。随后，我们引入了InstantStyle框架，通过实施两项关键策略来应对这些问题：1）一种直接机制，在特征空间内将风格与内容从参考图像中解耦，基于同一空间内的特征可相互加减的假设。2）仅将参考图像特征注入风格特定模块，从而防止风格泄露，并避免繁琐的权重调优，这在参数密集型设计中尤为常见。我们的工作展示了卓越的视觉风格化效果，在风格强度和文本元素可控性之间实现了最佳平衡。我们的代码将发布于https://github.com/InstantStyle/InstantStyle。

English

Tuning-free diffusion-based models have demonstrated significant potential in the realm of image personalization and customization. However, despite this notable progress, current models continue to grapple with several complex challenges in producing style-consistent image generation. Firstly, the concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure, among others. Secondly, inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. Lastly, adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability. In this paper, we commence by examining several compelling yet frequently overlooked observations. We then proceed to introduce InstantStyle, a framework designed to address these issues through the implementation of two key strategies: 1) A straightforward mechanism that decouples style and content from reference images within the feature space, predicated on the assumption that features within the same space can be either added to or subtracted from one another. 2) The injection of reference image features exclusively into style-specific blocks, thereby preventing style leaks and eschewing the need for cumbersome weight tuning, which often characterizes more parameter-heavy designs.Our work demonstrates superior visual stylization outcomes, striking an optimal balance between the intensity of style and the controllability of textual elements. Our codes will be available at https://github.com/InstantStyle/InstantStyle.