模塊化定制擴散模型的正交適應

摘要

對文本到圖像模型的定制技術為以往無法實現的廣泛應用打開了道路，使得能夠在不同背景和風格下生成特定概念成為可能。雖然現有方法能夠為個別概念或有限的預定義集合提供高保真度的定制，但在實現可擴展性方面仍有不足，即單個模型能夠無縫地呈現無數概念。本文提出了一個名為模塊化定制的新問題，旨在有效地合併為個別概念獨立進行微調的定制模型。這使得合併後的模型能夠共同合成一幅圖像中的概念，而不會影響保真度或增加任何額外的計算成本。為了解決這個問題，我們引入了正交適應，這是一種旨在鼓勵在微調期間互不訪問的定制模型具有正交殘差權重的方法。這確保在推斷時，定制模型可以被最小干擾地相加。我們提出的方法既簡單又多樣，適用於模型架構中幾乎所有可優化的權重。通過一系列定量和定性評估，我們的方法在效率和身份保留方面始終優於相關基準線，顯示出在擴散模型的可擴展定制方面取得了重大進展。

English

Customization techniques for text-to-image models have paved the way for a wide range of previously unattainable applications, enabling the generation of specific concepts across diverse contexts and styles. While existing methods facilitate high-fidelity customization for individual concepts or a limited, pre-defined set of them, they fall short of achieving scalability, where a single model can seamlessly render countless concepts. In this paper, we address a new problem called Modular Customization, with the goal of efficiently merging customized models that were fine-tuned independently for individual concepts. This allows the merged model to jointly synthesize concepts in one image without compromising fidelity or incurring any additional computational costs. To address this problem, we introduce Orthogonal Adaptation, a method designed to encourage the customized models, which do not have access to each other during fine-tuning, to have orthogonal residual weights. This ensures that during inference time, the customized models can be summed with minimal interference. Our proposed method is both simple and versatile, applicable to nearly all optimizable weights in the model architecture. Through an extensive set of quantitative and qualitative evaluations, our method consistently outperforms relevant baselines in terms of efficiency and identity preservation, demonstrating a significant leap toward scalable customization of diffusion models.

模塊化定制擴散模型的正交適應

Orthogonal Adaptation for Modular Customization of Diffusion Models

摘要

Support