SwapAnything：实现个性化视觉编辑中的任意对象交换

SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing

April 8, 2024

作者: Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang

cs.AI

摘要

在个人内容的有效编辑中扮演着关键角色，使个人能够表达创意，在视觉故事中编织引人入胜的叙事，并提升视觉内容的整体质量和影响力。因此，在这项工作中，我们介绍了SwapAnything，这是一个新颖的框架，可以用参考给出的个性化概念替换图像中的任何对象，同时保持上下文不变。与现有的个性化主题替换方法相比，SwapAnything具有三个独特优势：(1) 对任意对象和部分进行精确控制，而不是主要主题，(2) 更忠实地保留上下文像素，(3) 更好地将个性化概念适应于图像。首先，我们提出了有针对性的变量交换，通过对潜在特征图进行区域控制，并交换掩码变量以忠实地保留上下文并进行初始语义概念交换。然后，我们引入外观适应，以在图像生成过程中将语义概念无缝地调整到原始图像中，包括目标位置、形状、风格和内容。人类和自动评估的广泛结果显示，我们的方法在个性化交换方面显著优于基线方法。此外，SwapAnything展示了其在单个对象、多个对象、部分对象和跨领域交换任务中的精确和忠实交换能力。SwapAnything在基于文本的交换以及超越交换的任务，如对象插入，也取得了出色的性能。

English

Effective editing of personal content holds a pivotal role in enabling individuals to express their creativity, weaving captivating narratives within their visual stories, and elevate the overall quality and impact of their visual content. Therefore, in this work, we introduce SwapAnything, a novel framework that can swap any objects in an image with personalized concepts given by the reference, while keeping the context unchanged. Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image. First, we propose targeted variable swapping to apply region control over latent feature maps and swap masked variables for faithful context preservation and initial semantic concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt the semantic concept into the original image in terms of target location, shape, style, and content during the image generation process. Extensive results on both human and automatic evaluation demonstrate significant improvements of our approach over baseline methods on personalized swapping. Furthermore, SwapAnything shows its precise and faithful swapping abilities across single object, multiple objects, partial object, and cross-domain swapping tasks. SwapAnything also achieves great performance on text-based swapping and tasks beyond swapping such as object insertion.