TraDiffusion：基於軌跡的無需訓練的圖像生成

TraDiffusion: Trajectory-Based Training-Free Image Generation

August 19, 2024

作者: Mingrui Wu, Oucheng Huang, Jiayi Ji, Jiale Li, Xinyue Cai, Huafeng Kuang, Jianzhuang Liu, Xiaoshuai Sun, Rongrong Ji

cs.AI

摘要

在這項工作中，我們提出了一種無需訓練、基於軌跡可控的 T2I 方法，稱為 TraDiffusion。這一新穎方法使用戶可以輕鬆通過滑鼠軌跡引導圖像生成。為了實現精確控制，我們設計了一個距離感知能量函數，有效引導潛在變量，確保生成的焦點在軌跡定義的區域內。該能量函數包括一個控制函數，將生成物拉近到指定軌跡附近，以及一個移動函數，減少遠離軌跡的區域的活動。通過對 COCO 數據集進行廣泛實驗和定性評估，結果顯示 TraDiffusion 有助於更簡單、更自然的圖像控制。此外，它展示了在生成的圖像中操作突出區域、屬性和關係的能力，以及基於任意或增強軌跡的視覺輸入。

English

In this work, we propose a training-free, trajectory-based controllable T2I approach, termed TraDiffusion. This novel method allows users to effortlessly guide image generation via mouse trajectories. To achieve precise control, we design a distance awareness energy function to effectively guide latent variables, ensuring that the focus of generation is within the areas defined by the trajectory. The energy function encompasses a control function to draw the generation closer to the specified trajectory and a movement function to diminish activity in areas distant from the trajectory. Through extensive experiments and qualitative assessments on the COCO dataset, the results reveal that TraDiffusion facilitates simpler, more natural image control. Moreover, it showcases the ability to manipulate salient regions, attributes, and relationships within the generated images, alongside visual input based on arbitrary or enhanced trajectories.