用于减少扩散模型中曝光偏差的频谱先验

摘要

扩散模型在迭代采样过程中通常会出现误差累积问题，即所谓的“曝光偏差”。我们发现训练与推理之间存在系统性的频率依赖性偏差，这可解释为信噪比（SNR）随频率变化的误差。关键的是，这种失配的方向会随模型和时间步长而变化，表明固定修正规则不具备泛化能力。为此，我们提出频谱对齐（Spectral Alignment, SPA），这是一种轻量级、基于引导的方法，可将中间预测的功率谱校准至预计算的先验分布。该方法包含两个阶段：（1）基于训练数据离线拟合参数化频谱模型；（2）通过基于FFT的高效梯度计算实现推理时引导。SPA仅引入3-4%的额外计算开销，且与无分类器引导（CFG）互补。我们在多种架构上验证了其一致性改进，涵盖像素空间模型（DDPM、ADM）、潜在扩散模型（SD2.0、SDXL）以及流匹配模型（SD3.5、FLUX）。实现代码见https://github.com/SonyResearch/SPA。

English

Diffusion models typically suffer from error accumulation during iterative sampling, commonly referred to as exposure bias. We reveal systematic frequency-dependent discrepancies between training and inference, which can be interpreted as frequency-dependent SNR error. Crucially, the direction of this mismatch varies across models and timesteps, indicating that fixed correction rules do not generalize. We propose Spectral Alignment (SPA), a lightweight, guidance-based method that calibrates the power spectrum of intermediate predictions to a pre-computed prior. Our approach consists of two stages: (1) offline fitting of a parametric spectrum model from training data, and (2) inference-time guidance via efficient FFT-based gradient computation. SPA introduces minimal computational overhead (3-4\%) and is complementary to Classifier-Free Guidance (CFG). We demonstrate consistent improvements across diverse architectures, from pixel-space models (DDPM, ADM) to latent diffusion models (SD2.0, SDXL) and flow-matching models (SD3.5, FLUX). Our implementation is available at https://github.com/SonyResearch/SPA.