一個凍結的像素空間擴散模型可以用自身樣本進行自我引導

摘要

像素空間擴散模型旨在直接學習一個端到端生成器，作用於原始像素。這具有挑戰性，因為單一模型必須在同一高維空間中同時捕捉全域結構與局部紋理。雖然近期研究透過替代的預測目標、訓練目標與架構來改善像素擴散，但這些進展通常需要從零開始訓練新模型。我們展示了一種更便宜且互補的策略：一個凍結的、預訓練的像素擴散模型可以自我引導。我們的主要觀察是，預訓練的像素擴散 Transformer 的中間層可以被解碼為捕捉主要低頻結構的粗略預測，而最終層則逐步細化局部高頻細節。因此，我們將一個輕量級預測頭附加到中間層，保持骨幹網路凍結，並在採樣過程中，將中間預測與最終預測之間的差異用作自我引導方向。為了訓練這個預測頭，我們進一步發現真實影像並非必要；相反，模型生成的樣本就足夠，甚至在訓練預測頭時優於真實影像，特別是在增強像素擴散往往欠擬合的高頻分量方面。在 ImageNet 上的多個像素擴散模型中，我們的合成自我引導（SSG）持續改善生成質量，而適配器訓練所需的運算量不到完整模型訓練的 1%：在無分類器引導（CFG）下，它將評估的 JiT 變體的 FID 降低了超過 50%，並進一步改善了使用 CFG 的強基線，例如，將 JiT-H/16 從 1.86 降至 1.67，將 PixelREPA-H/16 從 1.81 降至 1.59。我們的程式碼可在 https://github.com/zfu006/SSG 取得。

English

Pixel-space diffusion models aim to learn an end-to-end generator directly over raw pixels. This is challenging because a single model must capture both global structure and local texture in the same high-dimensional space. While recent work improves pixel diffusion through alternative prediction targets, training objectives, and architectures, these advances typically require training a new model from scratch. We show there is a cheaper, complementary strategy: a frozen, pretrained pixel diffusion model can guide itself. Our key observation is that intermediate layers of a pretrained pixel diffusion transformer can be decoded into coarse predictions that capture the main low-frequency structure, while the final layers progressively refine local, high-frequency details. We therefore attach a lightweight prediction head to an intermediate layer, keep the backbone frozen, and use the discrepancy between the intermediate and final predictions as a self-guidance direction during sampling. To train this head, we further find that real images are not necessary. Instead, model-generated samples suffice and even outperform real images for training the head, especially in enhancing the high-frequency components that pixel diffusion tends to underfit. Across multiple pixel diffusion models on ImageNet, our Synthetic Self-Guidance (SSG) consistently improves generation while adapter training requires less than 1% of full-model training compute: it reduces FID by over 50% across the evaluated JiT variants without classifier-free guidance (CFG) and further improves strong baselines with CFG, e.g., JiT-H/16 from 1.86 to 1.67 and PixelREPA-H/16 from 1.81 to 1.59. Our code is available at https://github.com/zfu006/SSG.