DreamMatcher：外觀匹配自我關注，用於語義一致的文本到圖像個性化

DreamMatcher: Appearance Matching Self-Attention for Semantically-Consistent Text-to-Image Personalization

February 15, 2024

作者: Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, Seunggyu Chang

cs.AI

摘要

文本到圖像（T2I）個性化的目標是將擴散模型定制為用戶提供的參考概念，生成與目標提示對齊的概念的多樣圖像。傳統方法使用獨特的文本嵌入來表示參考概念，往往無法準確模仿參考的外觀。為了解決這個問題，一個解決方案可能是將參考圖像明確條件化到目標去噪過程中，這被稱為鍵值替換。然而，先前的工作受限於局部編輯，因為它們破壞了預訓練T2I模型的結構路徑。為了克服這一問題，我們提出了一種新的插件方法，稱為DreamMatcher，將T2I個性化重新定義為語義匹配。具體來說，DreamMatcher通過語義匹配將目標值替換為與之對齊的參考值，同時保持結構路徑不變，以保留預訓練T2I模型生成多樣結構的通用能力。我們還引入了一種語義一致的遮罩策略，以將個性化概念與目標提示引入的無關區域隔離開來。與現有的T2I模型兼容，DreamMatcher在複雜情境中顯示出顯著的改進。深入分析展示了我們方法的有效性。

English

The objective of text-to-image (T2I) personalization is to customize a diffusion model to a user-provided reference concept, generating diverse images of the concept aligned with the target prompts. Conventional methods representing the reference concepts using unique text embeddings often fail to accurately mimic the appearance of the reference. To address this, one solution may be explicitly conditioning the reference images into the target denoising process, known as key-value replacement. However, prior works are constrained to local editing since they disrupt the structure path of the pre-trained T2I model. To overcome this, we propose a novel plug-in method, called DreamMatcher, which reformulates T2I personalization as semantic matching. Specifically, DreamMatcher replaces the target values with reference values aligned by semantic matching, while leaving the structure path unchanged to preserve the versatile capability of pre-trained T2I models for generating diverse structures. We also introduce a semantic-consistent masking strategy to isolate the personalized concept from irrelevant regions introduced by the target prompts. Compatible with existing T2I models, DreamMatcher shows significant improvements in complex scenarios. Intensive analyses demonstrate the effectiveness of our approach.