3DGS-DET：为3D目标检测赋能的3D高斯飞溅技术，结合边界引导和以框为中心的采样。

3DGS-DET: Empower 3D Gaussian Splatting with Boundary Guidance and Box-Focused Sampling for 3D Object Detection

October 2, 2024

作者: Yang Cao, Yuanliang Jv, Dan Xu

cs.AI

摘要

神经辐射场（NeRF）被广泛用于新视角合成，并已被应用于三维物体检测（3DOD），为通过视图合成表示实现3DOD提供了一种有前途的方法。然而，NeRF面临固有限制：（i）由于其隐式性质，对于3DOD的表示能力有限，（ii）渲染速度较慢。最近，三维高斯斑点（3DGS）作为一种明确的三维表示出现，解决了这些限制。受到这些优势的启发，本文首次将3DGS引入3DOD，确定了两个主要挑战：（i）高斯斑点的模糊空间分布：3DGS主要依赖于2D像素级监督，导致高斯斑点的三维空间分布不清晰，对象与背景之间的区分不明显，这妨碍了3DOD；（ii）过多的背景斑点：2D图像通常包含大量背景像素，导致密集重建的3DGS具有许多代表背景的噪声高斯斑点，对检测产生负面影响。为了解决挑战（i），我们利用3DGS重建源自2D图像的事实，提出了一种优雅而高效的解决方案，即通过引入2D边界引导显著增强高斯斑点的空间分布，使对象与其背景之间的区分更加清晰。为了应对挑战（ii），我们提出了一种使用2D框的盒子聚焦采样策略，在3D空间中生成对象概率分布，允许在3D中进行有效的概率采样，保留更多对象斑点并减少嘈杂的背景斑点。受益于我们的设计，我们的3DGS-DET明显优于SOTA NeRF-based方法NeRF-Det，在ScanNet数据集上[email protected]提高了+6.6，[email protected]提高了+8.1，在ARKITScenes数据集上[email protected]提高了惊人的+31.5。

English

Neural Radiance Fields (NeRF) are widely used for novel-view synthesis and have been adapted for 3D Object Detection (3DOD), offering a promising approach to 3DOD through view-synthesis representation. However, NeRF faces inherent limitations: (i) limited representational capacity for 3DOD due to its implicit nature, and (ii) slow rendering speeds. Recently, 3D Gaussian Splatting (3DGS) has emerged as an explicit 3D representation that addresses these limitations. Inspired by these advantages, this paper introduces 3DGS into 3DOD for the first time, identifying two main challenges: (i) Ambiguous spatial distribution of Gaussian blobs: 3DGS primarily relies on 2D pixel-level supervision, resulting in unclear 3D spatial distribution of Gaussian blobs and poor differentiation between objects and background, which hinders 3DOD; (ii) Excessive background blobs: 2D images often include numerous background pixels, leading to densely reconstructed 3DGS with many noisy Gaussian blobs representing the background, negatively affecting detection. To tackle the challenge (i), we leverage the fact that 3DGS reconstruction is derived from 2D images, and propose an elegant and efficient solution by incorporating 2D Boundary Guidance to significantly enhance the spatial distribution of Gaussian blobs, resulting in clearer differentiation between objects and their background. To address the challenge (ii), we propose a Box-Focused Sampling strategy using 2D boxes to generate object probability distribution in 3D spaces, allowing effective probabilistic sampling in 3D to retain more object blobs and reduce noisy background blobs. Benefiting from our designs, our 3DGS-DET significantly outperforms the SOTA NeRF-based method, NeRF-Det, achieving improvements of +6.6 on [email protected] and +8.1 on [email protected] for the ScanNet dataset, and impressive +31.5 on [email protected] for the ARKITScenes dataset.