ObjectDrop：為逼真物體移除和插入提供反事實引導

ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertion

March 27, 2024

作者: Daniel Winter, Matan Cohen, Shlomi Fruchter, Yael Pritch, Alex Rav-Acha, Yedid Hoshen

cs.AI

摘要

擴散模型已經革新了影像編輯，但通常會生成違反物理法則的影像，特別是對場景中物體的影響，例如遮擋、陰影和反射效應。通過分析自監督方法的局限性，我們提出了一個實用解決方案，重點在於一個反事實數據集。我們的方法涉及在移除單個物體之前和之後捕獲場景，同時最小化其他變化。通過在這個數據集上微調擴散模型，我們不僅能夠移除物體，還能夠消除它們對場景的影響。然而，我們發現，將這種方法應用於逼真的物體插入需要一個不切實際地大的數據集。為了應對這一挑戰，我們提出了啟動監督方法；利用我們在一個小型反事實數據集上訓練的物體移除模型，我們可以大幅擴展這個數據集。我們的方法在逼真的物體移除和插入方面顯著優於先前的方法，特別是在對場景中物體影響的建模方面。

English

Diffusion models have revolutionized image editing but often generate images that violate physical laws, particularly the effects of objects on the scene, e.g., occlusions, shadows, and reflections. By analyzing the limitations of self-supervised approaches, we propose a practical solution centered on a counterfactual dataset. Our method involves capturing a scene before and after removing a single object, while minimizing other changes. By fine-tuning a diffusion model on this dataset, we are able to not only remove objects but also their effects on the scene. However, we find that applying this approach for photorealistic object insertion requires an impractically large dataset. To tackle this challenge, we propose bootstrap supervision; leveraging our object removal model trained on a small counterfactual dataset, we synthetically expand this dataset considerably. Our approach significantly outperforms prior methods in photorealistic object removal and insertion, particularly at modeling the effects of objects on the scene.