條件感知神經網絡用於受控圖像生成

Condition-Aware Neural Network for Controlled Image Generation

April 1, 2024

作者: Han Cai, Muyang Li, Zhuoyang Zhang, Qinsheng Zhang, Ming-Yu Liu, Song Han

cs.AI

摘要

我們提出了一種新的方法，稱為條件感知神經網絡（CAN），用於為圖像生成模型添加控制。與先前的條件控制方法平行，CAN通過動態調整神經網絡的權重來控制圖像生成過程。這是通過引入一個條件感知權重生成模塊來實現的，該模塊根據輸入條件為卷積/線性層生成條件權重。我們在ImageNet上進行了類條件圖像生成和在COCO上進行了文本到圖像生成的CAN測試。CAN在擴散變壓器模型（包括DiT和UViT）上始終提供顯著的改進。特別是，CAN與EfficientViT（CaT）在ImageNet 512x512上實現了2.78的FID，超越了DiT-XL/2，同時每個採樣步驟需要的MAC數量減少了52倍。

English

We present Condition-Aware Neural Network (CAN), a new method for adding control to image generative models. In parallel to prior conditional control methods, CAN controls the image generation process by dynamically manipulating the weight of the neural network. This is achieved by introducing a condition-aware weight generation module that generates conditional weight for convolution/linear layers based on the input condition. We test CAN on class-conditional image generation on ImageNet and text-to-image generation on COCO. CAN consistently delivers significant improvements for diffusion transformer models, including DiT and UViT. In particular, CAN combined with EfficientViT (CaT) achieves 2.78 FID on ImageNet 512x512, surpassing DiT-XL/2 while requiring 52x fewer MACs per sampling step.