Gamba：将高斯点光栅与Mamba相结合，用于单视角3D重建。

Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstruction

March 27, 2024

作者: Qiuhong Shen, Xuanyu Yi, Zike Wu, Pan Zhou, Hanwang Zhang, Shuicheng Yan, Xinchao Wang

cs.AI

摘要

我们面临着从单个图像高效重建3D资产的挑战，这是自动化3D内容创建流程需求不断增长的问题。先前的方法主要依赖于得分蒸馏采样（SDS）和神经辐射场（NeRF）。尽管这些方法取得了显著成功，但由于优化时间长和内存使用量大等实际限制，这些方法遇到了困难。在本报告中，我们介绍了Gamba，这是一个端到端摊销的3D重建模型，从单视图图像中重建，强调两个主要见解：（1）3D表示：利用大量3D高斯函数进行高效的3D高斯飞溅过程；（2）骨干设计：引入基于Mamba的顺序网络，促进依赖上下文的推理和与序列（令牌）长度的线性可伸缩性，适应大量高斯函数。Gamba整合了数据预处理、正则化设计和训练方法方面的重大进展。我们使用真实世界扫描的OmniObject3D数据集对Gamba进行了评估，与现有的基于优化和前馈的3D生成方法进行了比较。在这里，Gamba展示了竞争性的生成能力，无论是在质量上还是在数量上，同时在单个NVIDIA A100 GPU上实现了显著的速度，大约为0.6秒。

English

We tackle the challenge of efficiently reconstructing a 3D asset from a single image with growing demands for automated 3D content creation pipelines. Previous methods primarily rely on Score Distillation Sampling (SDS) and Neural Radiance Fields (NeRF). Despite their significant success, these approaches encounter practical limitations due to lengthy optimization and considerable memory usage. In this report, we introduce Gamba, an end-to-end amortized 3D reconstruction model from single-view images, emphasizing two main insights: (1) 3D representation: leveraging a large number of 3D Gaussians for an efficient 3D Gaussian splatting process; (2) Backbone design: introducing a Mamba-based sequential network that facilitates context-dependent reasoning and linear scalability with the sequence (token) length, accommodating a substantial number of Gaussians. Gamba incorporates significant advancements in data preprocessing, regularization design, and training methodologies. We assessed Gamba against existing optimization-based and feed-forward 3D generation approaches using the real-world scanned OmniObject3D dataset. Here, Gamba demonstrates competitive generation capabilities, both qualitatively and quantitatively, while achieving remarkable speed, approximately 0.6 second on a single NVIDIA A100 GPU.