EasyControl：為擴散變換器增添高效靈活的控制能力

EasyControl: Adding Efficient and Flexible Control for Diffusion Transformer

March 10, 2025

作者: Yuxuan Zhang, Yirui Yuan, Yiren Song, Haofan Wang, Jiaming Liu

cs.AI

摘要

基於Unet的擴散模型，如ControlNet和IP-Adapter，近期取得了顯著進展，引入了有效的空間與主體控制機制。然而，DiT（擴散變壓器）架構在實現高效且靈活的控制方面仍面臨挑戰。為解決這一問題，我們提出了EasyControl，這是一個旨在統一條件引導擴散變壓器的新框架，具備高效與靈活性。我們的框架基於三大創新點。首先，我們引入了一個輕量級的條件注入LoRA模塊。該模塊獨立處理條件信號，作為即插即用的解決方案，避免了修改基礎模型權重，確保了與定制模型的兼容性，並支持多樣化條件的靈活注入。值得注意的是，此模塊還支持和諧且穩健的零樣本多條件泛化，即便僅在單一條件數據上訓練。其次，我們提出了一種位置感知訓練範式。該方法將輸入條件標準化至固定分辨率，允許生成任意縱橫比和靈活分辨率的圖像，同時優化了計算效率，使框架更適合實際應用。第三，我們開發了一種結合KV緩存技術的因果注意力機制，專為條件生成任務設計。這一創新顯著降低了圖像合成的延遲，提升了框架的整體效率。通過大量實驗，我們證明了EasyControl在多種應用場景中均表現出色。這些創新共同使我們的框架高效、靈活，適用於廣泛的任務領域。

English

Recent advancements in Unet-based diffusion models, such as ControlNet and IP-Adapter, have introduced effective spatial and subject control mechanisms. However, the DiT (Diffusion Transformer) architecture still struggles with efficient and flexible control. To tackle this issue, we propose EasyControl, a novel framework designed to unify condition-guided diffusion transformers with high efficiency and flexibility. Our framework is built on three key innovations. First, we introduce a lightweight Condition Injection LoRA Module. This module processes conditional signals in isolation, acting as a plug-and-play solution. It avoids modifying the base model weights, ensuring compatibility with customized models and enabling the flexible injection of diverse conditions. Notably, this module also supports harmonious and robust zero-shot multi-condition generalization, even when trained only on single-condition data. Second, we propose a Position-Aware Training Paradigm. This approach standardizes input conditions to fixed resolutions, allowing the generation of images with arbitrary aspect ratios and flexible resolutions. At the same time, it optimizes computational efficiency, making the framework more practical for real-world applications. Third, we develop a Causal Attention Mechanism combined with the KV Cache technique, adapted for conditional generation tasks. This innovation significantly reduces the latency of image synthesis, improving the overall efficiency of the framework. Through extensive experiments, we demonstrate that EasyControl achieves exceptional performance across various application scenarios. These innovations collectively make our framework highly efficient, flexible, and suitable for a wide range of tasks.