ConceptLab：使用擴散先驗約束進行創意生成

ConceptLab: Creative Generation using Diffusion Prior Constraints

August 3, 2023

作者: Elad Richardson, Kfir Goldberg, Yuval Alaluf, Daniel Cohen-Or

cs.AI

摘要

最近的文本轉圖像生成模型使我們能夠將我們的文字轉換為生動迷人的圖像。隨之而來的個性化技術激增也使我們能夠在新場景中想像獨特的概念。然而，一個耐人尋味的問題仍然存在：我們如何生成一個從未被看過的新奇概念？在本文中，我們提出了創意文本轉圖像生成的任務，我們試圖生成廣泛類別的新成員（例如，生成一種與所有現有寵物不同的寵物）。我們利用鮮為人知的擴散先驗模型，並展示創造性生成問題可以被制定為在擴散先驗輸出空間上的優化過程，產生一組“先驗約束”。為了使我們生成的概念不會收斂為現有成員，我們將一個問答模型納入其中，自適應地向優化問題添加新約束，鼓勵模型發現越來越獨特的創作。最後，我們展示了我們的先驗約束也可以作為一個強大的混合機制，使我們能夠創建生成概念之間的混合體，為創意過程引入更多靈活性。

English

Recent text-to-image generative models have enabled us to transform our words into vibrant, captivating imagery. The surge of personalization techniques that has followed has also allowed us to imagine unique concepts in new scenes. However, an intriguing question remains: How can we generate a new, imaginary concept that has never been seen before? In this paper, we present the task of creative text-to-image generation, where we seek to generate new members of a broad category (e.g., generating a pet that differs from all existing pets). We leverage the under-studied Diffusion Prior models and show that the creative generation problem can be formulated as an optimization process over the output space of the diffusion prior, resulting in a set of "prior constraints". To keep our generated concept from converging into existing members, we incorporate a question-answering model that adaptively adds new constraints to the optimization problem, encouraging the model to discover increasingly more unique creations. Finally, we show that our prior constraints can also serve as a strong mixing mechanism allowing us to create hybrids between generated concepts, introducing even more flexibility into the creative process.