SwapAnything：在個性化視覺編輯中實現任意物件交換

SwapAnything: Enabling Arbitrary Object Swapping in Personalized Visual Editing

April 8, 2024

作者: Jing Gu, Yilin Wang, Nanxuan Zhao, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, Xin Eric Wang

cs.AI

摘要

有效編輯個人內容在幫助個人表達創意、在視覺故事中編織引人入勝的敘述，以及提升視覺內容的整體質量和影響力方面扮演著關鍵角色。因此，在這項工作中，我們介紹了一個名為SwapAnything的新框架，該框架可以將圖像中的任何物件與參考中給定的個性化概念進行交換，同時保持上下文不變。與現有的個性化主題交換方法相比，SwapAnything具有三個獨特優勢：(1) 對任意物件和部分進行精確控制，而不僅僅是主題，(2) 更忠實地保留上下文像素，(3) 更好地將個性化概念適應於圖像。首先，我們提出了針對性的變量交換，以在潛在特徵圖上應用區域控制，並交換遮罩變量以實現忠實的上下文保留和初始語義概念交換。然後，我們引入外觀適應，以在圖像生成過程中無縫地將語義概念調整到原始圖像中，包括目標位置、形狀、風格和內容。人類和自動評估的廣泛結果顯示，我們的方法在個性化交換方面顯著優於基準方法。此外，SwapAnything展示了其在單個物件、多個物件、部分物件和跨領域交換任務中的精確和忠實的交換能力。SwapAnything在基於文本的交換以及超越交換的任務，如物件插入方面也取得了出色的表現。

English

Effective editing of personal content holds a pivotal role in enabling individuals to express their creativity, weaving captivating narratives within their visual stories, and elevate the overall quality and impact of their visual content. Therefore, in this work, we introduce SwapAnything, a novel framework that can swap any objects in an image with personalized concepts given by the reference, while keeping the context unchanged. Compared with existing methods for personalized subject swapping, SwapAnything has three unique advantages: (1) precise control of arbitrary objects and parts rather than the main subject, (2) more faithful preservation of context pixels, (3) better adaptation of the personalized concept to the image. First, we propose targeted variable swapping to apply region control over latent feature maps and swap masked variables for faithful context preservation and initial semantic concept swapping. Then, we introduce appearance adaptation, to seamlessly adapt the semantic concept into the original image in terms of target location, shape, style, and content during the image generation process. Extensive results on both human and automatic evaluation demonstrate significant improvements of our approach over baseline methods on personalized swapping. Furthermore, SwapAnything shows its precise and faithful swapping abilities across single object, multiple objects, partial object, and cross-domain swapping tasks. SwapAnything also achieves great performance on text-based swapping and tasks beyond swapping such as object insertion.