D-Flow:通過流動進行控制生成
D-Flow: Differentiating through Flows for Controlled Generation
February 21, 2024
作者: Heli Ben-Hamu, Omri Puny, Itai Gat, Brian Karrer, Uriel Singer, Yaron Lipman
cs.AI
摘要
在無需重新訓練特定任務模型的情況下,控制最先進的擴散和流匹配(FM)模型生成結果的能力,為解決反問題、有條件生成以及一般控制生成提供了一個強大的工具。在這項工作中,我們介紹了D-Flow,一個簡單的框架,通過對流進行微分,優化源(噪聲)點來控制生成過程。我們通過關鍵觀察來推動這個框架,該觀察指出,對於使用高斯概率路徑訓練的擴散/FM模型,通過生成過程進行微分會將梯度投影到數據流形上,從而將先驗隱式注入到優化過程中。我們在線性和非線性控制生成問題上驗證了我們的框架,包括:圖像和音頻反問題以及有條件的分子生成,在所有問題上均達到了最先進的性能。
English
Taming the generation outcome of state of the art Diffusion and Flow-Matching
(FM) models without having to re-train a task-specific model unlocks a powerful
tool for solving inverse problems, conditional generation, and controlled
generation in general. In this work we introduce D-Flow, a simple framework for
controlling the generation process by differentiating through the flow,
optimizing for the source (noise) point. We motivate this framework by our key
observation stating that for Diffusion/FM models trained with Gaussian
probability paths, differentiating through the generation process projects
gradient on the data manifold, implicitly injecting the prior into the
optimization process. We validate our framework on linear and non-linear
controlled generation problems including: image and audio inverse problems and
conditional molecule generation reaching state of the art performance across
all.