ObjectDrop：为逼真物体移除和插入引导式反事实生成对抗训练

ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion

March 27, 2024

作者: Daniel Winter, Matan Cohen, Shlomi Fruchter, Yael Pritch, Alex Rav-Acha, Yedid Hoshen

cs.AI

摘要

扩散模型已经彻底改变了图像编辑，但通常会生成违反物理定律的图像，特别是对象对场景的影响，比如遮挡、阴影和反射效应。通过分析自监督方法的局限性，我们提出了一个围绕反事实数据集的实用解决方案。我们的方法涉及在移除单个对象之前和之后捕获场景，同时最小化其他变化。通过在该数据集上微调扩散模型，我们不仅能够移除对象，还能消除它们对场景的影响。然而，我们发现将这种方法应用于逼真的对象插入需要一个不切实际大的数据集。为了应对这一挑战，我们提出了引导监督；利用我们在小规模反事实数据集上训练的对象移除模型，我们大幅度地扩展了这个数据集。我们的方法在逼真的对象移除和插入方面明显优于先前的方法，特别是在建模对象对场景的影响方面。

English

Diffusion models have revolutionized image editing but often generate images that violate physical laws, particularly the effects of objects on the scene, e.g., occlusions, shadows, and reflections. By analyzing the limitations of self-supervised approaches, we propose a practical solution centered on a counterfactual dataset. Our method involves capturing a scene before and after removing a single object, while minimizing other changes. By fine-tuning a diffusion model on this dataset, we are able to not only remove objects but also their effects on the scene. However, we find that applying this approach for photorealistic object insertion requires an impractically large dataset. To tackle this challenge, we propose bootstrap supervision; leveraging our object removal model trained on a small counterfactual dataset, we synthetically expand this dataset considerably. Our approach significantly outperforms prior methods in photorealistic object removal and insertion, particularly at modeling the effects of objects on the scene.