可微分的方块世界：通过渲染基元进行定性三维分解

Differentiable Blocks World: Qualitative 3D Decomposition by Rendering Primitives

July 11, 2023

作者: Tom Monnier, Jake Austin, Angjoo Kanazawa, Alexei A. Efros, Mathieu Aubry

cs.AI

摘要

在给定一组校准图像的情况下，我们提出了一种方法，通过3D基元生成简单、紧凑且可操作的3D世界表示。虽然许多方法侧重于恢复高保真度的3D场景，但我们专注于将场景解析为由少量纹理基元组成的中级3D表示。这种表示具有可解释性，易于操作，并适用于基于物理的模拟。此外，与现有的基元分解方法依赖于3D输入数据不同，我们的方法通过可微渲染直接在图像上操作。具体而言，我们将基元建模为纹理超四面体网格，并通过图像渲染损失从头开始优化它们的参数。我们强调为每个基元建模透明度的重要性，这对优化至关重要，同时也能处理不同数量的基元。我们展示了由纹理基元重建输入图像并准确建模可见的3D点，同时提供未见物体区域的全模态形状补全。我们将我们的方法与来自DTU的各种场景的最新技术进行了比较，并展示了它在来自BlendedMVS和Nerfstudio的现实捕获中的稳健性。我们还展示了如何利用我们的结果轻松编辑场景或执行物理模拟。代码和视频结果可在https://www.tmonnier.com/DBW 获取。

English

Given a set of calibrated images of a scene, we present an approach that produces a simple, compact, and actionable 3D world representation by means of 3D primitives. While many approaches focus on recovering high-fidelity 3D scenes, we focus on parsing a scene into mid-level 3D representations made of a small set of textured primitives. Such representations are interpretable, easy to manipulate and suited for physics-based simulations. Moreover, unlike existing primitive decomposition methods that rely on 3D input data, our approach operates directly on images through differentiable rendering. Specifically, we model primitives as textured superquadric meshes and optimize their parameters from scratch with an image rendering loss. We highlight the importance of modeling transparency for each primitive, which is critical for optimization and also enables handling varying numbers of primitives. We show that the resulting textured primitives faithfully reconstruct the input images and accurately model the visible 3D points, while providing amodal shape completions of unseen object regions. We compare our approach to the state of the art on diverse scenes from DTU, and demonstrate its robustness on real-life captures from BlendedMVS and Nerfstudio. We also showcase how our results can be used to effortlessly edit a scene or perform physical simulations. Code and video results are available at https://www.tmonnier.com/DBW .