DiffusionBench：論擴散Transformer的整體評估

摘要

擴散變換器 (DiT) 在影像生成方面的研究已收斂至單一評估設定：在 ImageNet 上進行類別條件生成。儘管各種方法改進了 FID 及相關指標，但這些指標是否真正反映生成建模的實質進展，已變得越來越不明確。自然的替代方案，即文字到影像 (T2I) 生成，卻被認為訓練與評估成本過高或不方便，因而經常被忽略。我們主張此觀點已不再成立。我們推出 NanoGen，一個統一的 DiT 訓練與評估框架。NanoGen 在 ImageNet 上與最先進的 DiT 基準線相匹配，且僅需更改 12 行設定，即可訓練出具競爭力的文字到影像模型。它目前支援在 ImageNet 與 T2I 設定下的 RAE、VAE、像素空間及 MeanFlow 擴散方法。在 NanoGen 下，訓練 T2I 所需的運算量與 ImageNet 相當。在利用 NanoGen 訓練 21 個潛在擴散模型後，我們觀察到方法排名在 ImageNet 與 T2I 生成之間並無強相關性：三項指標的皮爾遜相關係數介於 -0.377 至 -0.580 之間。這表明，一個能改善類別條件 ImageNet FID 的方法，在 T2I 上可能並無相應的改進，清楚說明了在兩項任務上評估 DiT 的必要性。為此，我們彙整了 ImageNet 與文字到影像的結果，形成了 DiffusionBench，一個全面的 DiT 研究基準。我們建議以 DiffusionBench 取代僅報告 ImageNet 的做法：能夠改善 DiffusionBench 的方法，更可能反映出廣泛的進展。

English

Diffusion transformer (DiT) research on image generation has converged to a single evaluation setup: class-conditional generation on ImageNet. While methods improve the FID and related metrics, it is increasingly unclear whether they reflect real progress in generative modeling. The natural alternative, i.e., text-to-image (T2I) generation, is perceived as too costly or inconvenient to train and evaluate and is often skipped. We argue that this perception no longer holds. We introduce NanoGen, a unified DiT training and evaluation framework. NanoGen matches state-of-the-art DiT baselines on ImageNet and, with 12 lines of configuration change, also trains competitive text-to-image models. It currently supports RAE, VAE, pixel-space, and MeanFlow diffusion methods under both ImageNet and T2I setups. Under NanoGen, training T2I requires comparable compute to ImageNet. After training 21 latent diffusion models with NanoGen, we observe that method ranking shows no strong correlation between ImageNet and T2I generation: Pearson correlation is between -0.377 and -0.580 across three metrics. This suggests that a method which improves class-conditional ImageNet FID may show no corresponding improvement on T2I, clearly indicating the necessity of evaluating DiTs on both tasks. To this end, we summarize ImageNet and text-to-image results, which yields DiffusionBench, a holistic benchmark for DiT research. We recommend reporting DiffusionBench in place of ImageNet alone: methods that improve DiffusionBench are more likely to reflect broader progress.