TraDiffusion：基于轨迹的无需训练的图像生成

TraDiffusion: Trajectory-Based Training-Free Image Generation

August 19, 2024

作者: Mingrui Wu, Oucheng Huang, Jiayi Ji, Jiale Li, Xinyue Cai, Huafeng Kuang, Jianzhuang Liu, Xiaoshuai Sun, Rongrong Ji

cs.AI

摘要

在这项工作中，我们提出了一种无需训练、基于轨迹可控的T2I方法，称为TraDiffusion。这种新颖的方法允许用户通过鼠标轨迹轻松引导图像生成。为了实现精确控制，我们设计了一个距离感知能量函数，有效地引导潜在变量，确保生成的焦点位于轨迹定义的区域内。能量函数包括一个控制函数，将生成物靠近指定轨迹，以及一个移动函数，减少远离轨迹的区域的活动。通过在COCO数据集上进行大量实验和定性评估，结果显示TraDiffusion有助于更简单、更自然的图像控制。此外，它展示了在生成的图像中操纵显著区域、属性和关系的能力，以及基于任意或增强轨迹的视觉输入。

English

In this work, we propose a training-free, trajectory-based controllable T2I approach, termed TraDiffusion. This novel method allows users to effortlessly guide image generation via mouse trajectories. To achieve precise control, we design a distance awareness energy function to effectively guide latent variables, ensuring that the focus of generation is within the areas defined by the trajectory. The energy function encompasses a control function to draw the generation closer to the specified trajectory and a movement function to diminish activity in areas distant from the trajectory. Through extensive experiments and qualitative assessments on the COCO dataset, the results reveal that TraDiffusion facilitates simpler, more natural image control. Moreover, it showcases the ability to manipulate salient regions, attributes, and relationships within the generated images, alongside visual input based on arbitrary or enhanced trajectories.