使用扩散模型解决反问题的变分视角

A Variational Perspective on Solving Inverse Problems with Diffusion Models

May 7, 2023

作者: Morteza Mardani, Jiaming Song, Jan Kautz, Arash Vahdat

cs.AI

摘要

扩散模型已经成为视觉领域基础模型中的关键支柱之一。它们的一个关键应用是通过单一扩散先验普遍解决不同下游逆任务，而无需为每个任务重新训练。大多数逆任务可以被表述为推断给定测量（例如，遮罩图像）的数据后验分布（例如，完整图像）。然而，在扩散模型中，由于扩散过程的非线性和迭代特性使得后验难以处理，这是一个挑战。为了应对这一挑战，我们提出了一种变分方法，通过设计寻求逼近真实后验分布。我们展示了我们的方法自然地导致通过去噪扩散过程（RED-Diff）进行正则化，其中不同时间步的去噪器同时对图像施加不同的结构约束。为了衡量来自不同时间步的去噪器的贡献，我们提出了基于信噪比（SNR）的加权机制。我们的方法为使用扩散模型解决逆问题提供了新的变分视角，允许我们将采样公式化为随机优化，从而可以简单地应用具有轻量级迭代的现成求解器。我们针对图像修复任务（如修补和超分辨率）的实验展示了我们的方法相对于基于采样的扩散模型的最新技术的优势。

English

Diffusion models have emerged as a key pillar of foundation models in visual domains. One of their critical applications is to universally solve different downstream inverse tasks via a single diffusion prior without re-training for each task. Most inverse tasks can be formulated as inferring a posterior distribution over data (e.g., a full image) given a measurement (e.g., a masked image). This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. To cope with this challenge, we propose a variational approach that by design seeks to approximate the true posterior distribution. We show that our approach naturally leads to regularization by denoising diffusion process (RED-Diff) where denoisers at different timesteps concurrently impose different structural constraints over the image. To gauge the contribution of denoisers from different timesteps, we propose a weighting mechanism based on signal-to-noise-ratio (SNR). Our approach provides a new variational perspective for solving inverse problems with diffusion models, allowing us to formulate sampling as stochastic optimization, where one can simply apply off-the-shelf solvers with lightweight iterates. Our experiments for image restoration tasks such as inpainting and superresolution demonstrate the strengths of our method compared with state-of-the-art sampling-based diffusion models.