LooseControl：提升ControlNet以進行廣義深度條件控制

LooseControl: Lifting ControlNet for Generalized Depth Conditioning

December 5, 2023

作者: Shariq Farooq Bhat, Niloy J. Mitra, Peter Wonka

cs.AI

摘要

我們提出LooseControl，以允許擴展式深度條件控制用於基於擴散的圖像生成。ControlNet，作為深度條件圖像生成的最先進技術，產生了卓越的結果，但依賴於對詳細深度圖的訪問。在許多情況下，創建這樣精確的深度圖是具有挑戰性的。本文介紹了一個通用版本的深度條件控制，可以啟用許多新的內容創建工作流程。具體而言，我們允許（C1）場景邊界控制，用於僅通過邊界條件粗略指定場景，以及（C2）3D框控制，用於指定目標對象的佈局位置，而不是對象的確切形狀和外觀。使用LooseControl，連同文本指導，用戶可以通過僅指定場景邊界和主要對象的位置來創建複雜的環境（例如房間，街景等）。此外，我們提供兩種編輯機制來完善結果：（E1）3D框編輯使用戶可以通過更改、添加或刪除框來完善圖像，同時凍結圖像的風格。這除了由編輯的框引起的變化外，幾乎沒有其他變化。 （E2）屬性編輯提出了可能的編輯方向，以更改場景的某個特定方面，例如整體對象密度或特定對象。通過廣泛的測試和與基準的比較，證明了我們方法的通用性。我們相信LooseControl可以成為一個重要的設計工具，用於輕鬆創建複雜環境，並可擴展到其他形式的引導通道。代碼和更多信息可在https://shariqfarooq123.github.io/loose-control/ 上找到。

English

We present LooseControl to allow generalized depth conditioning for diffusion-based image generation. ControlNet, the SOTA for depth-conditioned image generation, produces remarkable results but relies on having access to detailed depth maps for guidance. Creating such exact depth maps, in many scenarios, is challenging. This paper introduces a generalized version of depth conditioning that enables many new content-creation workflows. Specifically, we allow (C1) scene boundary control for loosely specifying scenes with only boundary conditions, and (C2) 3D box control for specifying layout locations of the target objects rather than the exact shape and appearance of the objects. Using LooseControl, along with text guidance, users can create complex environments (e.g., rooms, street views, etc.) by specifying only scene boundaries and locations of primary objects. Further, we provide two editing mechanisms to refine the results: (E1) 3D box editing enables the user to refine images by changing, adding, or removing boxes while freezing the style of the image. This yields minimal changes apart from changes induced by the edited boxes. (E2) Attribute editing proposes possible editing directions to change one particular aspect of the scene, such as the overall object density or a particular object. Extensive tests and comparisons with baselines demonstrate the generality of our method. We believe that LooseControl can become an important design tool for easily creating complex environments and be extended to other forms of guidance channels. Code and more information are available at https://shariqfarooq123.github.io/loose-control/ .