InstantStyle：在文本到圖像生成中朝向保留風格的自由午餐

InstantStyle: Free Lunch towards Style-Preserving in Text-to-Image Generation

April 3, 2024

作者: Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, Anthony Chen

cs.AI

摘要

無需調整的擴散式模型在圖像個性化和定制領域展現了顯著的潛力。然而，儘管取得了顯著進展，當前模型仍然面臨著在生成風格一致圖像方面的幾個複雜挑戰。首先，風格的概念本質上是不確定的，包括諸如顏色、材料、氛圍、設計和結構等多個元素。其次，基於反演的方法容易出現風格退化問題，通常導致細節的喪失。最後，基於適配器的方法經常需要對每個參考圖像進行細緻的權重調整，以實現風格強度和文本可控性之間的平衡。本文首先檢視幾個引人注目但經常被忽視的觀察，然後介紹InstantStyle，這是一個旨在通過實施兩個關鍵策略來解決這些問題的框架：1）一個簡單的機制，將風格和內容從特徵空間中的參考圖像中解耦，基於這樣一個假設，即同一空間中的特徵可以相互添加或相互減去。2）將參考圖像特徵專門注入到風格特定塊中，從而防止風格洩漏，避免繁瑣的權重調整，這經常是更多參數密集型設計的特徵。我們的工作展示了卓越的視覺風格化結果，實現了風格強度與文本元素可控性之間的最佳平衡。我們的代碼將在https://github.com/InstantStyle/InstantStyle 上提供。

English

Tuning-free diffusion-based models have demonstrated significant potential in the realm of image personalization and customization. However, despite this notable progress, current models continue to grapple with several complex challenges in producing style-consistent image generation. Firstly, the concept of style is inherently underdetermined, encompassing a multitude of elements such as color, material, atmosphere, design, and structure, among others. Secondly, inversion-based methods are prone to style degradation, often resulting in the loss of fine-grained details. Lastly, adapter-based approaches frequently require meticulous weight tuning for each reference image to achieve a balance between style intensity and text controllability. In this paper, we commence by examining several compelling yet frequently overlooked observations. We then proceed to introduce InstantStyle, a framework designed to address these issues through the implementation of two key strategies: 1) A straightforward mechanism that decouples style and content from reference images within the feature space, predicated on the assumption that features within the same space can be either added to or subtracted from one another. 2) The injection of reference image features exclusively into style-specific blocks, thereby preventing style leaks and eschewing the need for cumbersome weight tuning, which often characterizes more parameter-heavy designs.Our work demonstrates superior visual stylization outcomes, striking an optimal balance between the intensity of style and the controllability of textual elements. Our codes will be available at https://github.com/InstantStyle/InstantStyle.