LRM-Zero：使用合成数据训练大型重建模型

LRM-Zero: Training Large Reconstruction Models with Synthesized Data

June 13, 2024

作者: Desai Xie, Sai Bi, Zhixin Shu, Kai Zhang, Zexiang Xu, Yi Zhou, Sören Pirk, Arie Kaufman, Xin Sun, Hao Tan

cs.AI

摘要

我们提出了LRM-Zero，这是一个完全基于合成的3D数据训练的大型重建模型（LRM），实现了高质量的稀疏视角3D重建。LRM-Zero的核心是我们的程序化3D数据集Zeroverse，它是从简单的基本形状自动生成的，具有随机纹理和增强（例如，高度场、布尔差异和线框）。与先前的3D数据集（例如Objaverse）不同，它们通常是由人类捕获或制作以逼真地逼近真实3D数据的情况不同，Zeroverse完全忽略了逼真的全局语义，但在几何和纹理细节方面非常丰富，这些细节在局部上与真实对象相似甚至更为复杂。我们展示了我们的LRM-Zero，利用我们完全合成的Zeroverse进行训练，可以在重建真实世界对象时实现高视觉质量，与在Objaverse上训练的模型相媲美。我们还分析了Zeroverse的几个关键设计选择，这些选择有助于LRM-Zero的能力和训练稳定性。我们的工作表明，在3D视觉中的核心任务之一——3D重建，有可能在不考虑真实世界对象语义的情况下进行处理。Zeroverse的程序化合成代码和交互式可视化可在以下网址找到：https://desaixie.github.io/lrm-zero/。

English

We present LRM-Zero, a Large Reconstruction Model (LRM) trained entirely on synthesized 3D data, achieving high-quality sparse-view 3D reconstruction. The core of LRM-Zero is our procedural 3D dataset, Zeroverse, which is automatically synthesized from simple primitive shapes with random texturing and augmentations (e.g., height fields, boolean differences, and wireframes). Unlike previous 3D datasets (e.g., Objaverse) which are often captured or crafted by humans to approximate real 3D data, Zeroverse completely ignores realistic global semantics but is rich in complex geometric and texture details that are locally similar to or even more intricate than real objects. We demonstrate that our LRM-Zero, trained with our fully synthesized Zeroverse, can achieve high visual quality in the reconstruction of real-world objects, competitive with models trained on Objaverse. We also analyze several critical design choices of Zeroverse that contribute to LRM-Zero's capability and training stability. Our work demonstrates that 3D reconstruction, one of the core tasks in 3D vision, can potentially be addressed without the semantics of real-world objects. The Zeroverse's procedural synthesis code and interactive visualization are available at: https://desaixie.github.io/lrm-zero/.