排斥分数蒸馏用于扩展扩散模型的多样抽样

Repulsive Score Distillation for Diverse Sampling of Diffusion Models

June 24, 2024

作者: Nicolas Zilberstein, Morteza Mardani, Santiago Segarra

cs.AI

摘要

得分蒸馏采样对于将扩散模型整合到复杂视觉生成中至关重要。尽管取得了令人印象深刻的结果，但存在模式崩溃和缺乏多样性的问题。为了应对这一挑战，我们利用得分蒸馏的梯度流解释提出了排斥式得分蒸馏（RSD）。具体而言，我们提出了一个基于粒子集合的排斥变分框架，促进多样性。通过包含粒子之间耦合的变分近似，排斥表现为一种简单的正则化，允许基于它们的相对成对相似性进行粒子交互，例如通过径向基核进行测量。我们设计了适用于无约束和约束采样场景的RSD。对于约束采样，我们专注于潜空间中的反问题，导致增广变分公式，实现了在计算、质量和多样性之间的良好平衡。我们进行了大量实验，针对文本到图像生成和反问题，证明了RSD相对于最先进的替代方案在多样性和质量之间实现了卓越的权衡。

English

Score distillation sampling has been pivotal for integrating diffusion models into generation of complex visuals. Despite impressive results it suffers from mode collapse and lack of diversity. To cope with this challenge, we leverage the gradient flow interpretation of score distillation to propose Repulsive Score Distillation (RSD). In particular, we propose a variational framework based on repulsion of an ensemble of particles that promotes diversity. Using a variational approximation that incorporates a coupling among particles, the repulsion appears as a simple regularization that allows interaction of particles based on their relative pairwise similarity, measured e.g., via radial basis kernels. We design RSD for both unconstrained and constrained sampling scenarios. For constrained sampling we focus on inverse problems in the latent space that leads to an augmented variational formulation, that strikes a good balance between compute, quality and diversity. Our extensive experiments for text-to-image generation, and inverse problems demonstrate that RSD achieves a superior trade-off between diversity and quality compared with state-of-the-art alternatives.