反映现实：使扩散模型能够产生忠实的镜像反射

Reflecting Reality: Enabling Diffusion Models to Produce Faithful Mirror Reflections

September 23, 2024

作者: Ankit Dhiman, Manan Shah, Rishubh Parihar, Yash Bhalgat, Lokesh R Boregowda, R Venkatesh Babu

cs.AI

摘要

我们解决了使用基于扩散的生成模型生成高度逼真和可信的镜像反射的问题。我们将这个问题表述为图像修复任务，允许用户在生成过程中更好地控制镜子的放置。为了实现这一点，我们创建了SynMirror，这是一个大规模数据集，包含了各种合成场景，场景中的物体放置在镜子前面。SynMirror包含了大约198K个样本，从66K个独特的3D物体渲染而成，还包括它们的深度图、法线图和实例分割掩模，以捕捉场景的相关几何属性。利用这个数据集，我们提出了一种新颖的深度条件修复方法，名为MirrorFusion，它可以在给定输入图像和描绘镜子区域的掩模的情况下生成高质量、几何一致和照片逼真的镜像反射。通过广泛的定量和定性分析，MirrorFusion在SynMirror上表现优于最先进的方法。据我们所知，我们是第一个成功解决使用基于扩散的模型生成场景中对象的受控和忠实的镜像反射这一具有挑战性的问题的团队。SynMirror和MirrorFusion为从业者和研究人员开辟了图像编辑和增强现实应用的新途径。

English

We tackle the problem of generating highly realistic and plausible mirror reflections using diffusion-based generative models. We formulate this problem as an image inpainting task, allowing for more user control over the placement of mirrors during the generation process. To enable this, we create SynMirror, a large-scale dataset of diverse synthetic scenes with objects placed in front of mirrors. SynMirror contains around 198K samples rendered from 66K unique 3D objects, along with their associated depth maps, normal maps and instance-wise segmentation masks, to capture relevant geometric properties of the scene. Using this dataset, we propose a novel depth-conditioned inpainting method called MirrorFusion, which generates high-quality geometrically consistent and photo-realistic mirror reflections given an input image and a mask depicting the mirror region. MirrorFusion outperforms state-of-the-art methods on SynMirror, as demonstrated by extensive quantitative and qualitative analysis. To the best of our knowledge, we are the first to successfully tackle the challenging problem of generating controlled and faithful mirror reflections of an object in a scene using diffusion based models. SynMirror and MirrorFusion open up new avenues for image editing and augmented reality applications for practitioners and researchers alike.