冻结像素空间扩散模型可利用自身样本实现自我引导

摘要

像素空间扩散模型旨在直接在原始像素上学习端到端生成器。这颇具挑战性，因为单个模型必须在同一高维空间中同时捕获全局结构和局部纹理。尽管近期工作通过替代预测目标、训练目标和架构改进了像素扩散模型，但这些进展通常需要从头训练新模型。我们展示了一种更廉价且互补的策略：冻结的预训练像素扩散模型可以自我引导。我们的关键观察是，预训练像素扩散Transformer的中间层可以解码为捕获主要低频结构的粗糙预测，而最终层则逐步细化局部高频细节。因此，我们将轻量级预测头附加到中间层，保持主干网络冻结，并在采样过程中将中间预测与最终预测之间的差异用作自引导方向。为训练该预测头，我们进一步发现真实图像并非必需。相反，模型生成的样本足以训练该预测头，甚至在增强像素扩散往往欠拟合的高频分量方面优于真实图像。在ImageNet上的多个像素扩散模型中，我们的合成自引导（SSG）持续提升生成质量，而适配器训练所需计算量不到全模型训练计算量的1%：在无分类器引导（CFG）的情况下，它使所评估的JiT变体的FID降低超过50%，并进一步改进了使用CFG的强基线，例如JiT-H/16从1.86降至1.67，PixelREPA-H/16从1.81降至1.59。我们的代码可在https://github.com/zfu006/SSG获取。

English

Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels. This is challenging because a single model must capture both global structure and local texture in the same high-dimensional space. While recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures, these advances typically require training a new model from scratch. We show there is a cheaper, complementary strategy: a frozen, pretrained pixel diffusion model can guide itself. Our key observation is that intermediate layers of a pretrained pixel diffusion transformer can be decoded into coarse predictions that capture the main low-frequency structure, while the final layers progressively refine local, high-frequency details. We therefore attach a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling. To train this head, we further find that real images are not necessary. Instead, model-generated samples suffice and even outperform real images for training the head, especially in enhancing the high-frequency components that pixel diffusion tends to underfit. Across multiple pixel diffusion models on ImageNet, our Synthetic Self-Guidance (SSG) consistently improves generation while adapter training requires less than 1% of full-model training compute: it reduces FID by over 50% across the evaluated JiT variants without classifier-free guidance (CFG) and further improves strong baselines with CFG, e.g., JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59. Our code is available at https://github.com/zfu006/SSG.