具有神經補償的光譜修剪高斯場

Spectrally Pruned Gaussian Fields with Neural Compensation

May 1, 2024

作者: Runyi Yang, Zhenxin Zhu, Zhou Jiang, Baijun Ye, Xiaoxue Chen, Yifei Zhang, Yuantao Chen, Jian Zhao, Hao Zhao

cs.AI

摘要

最近，作為一種新穎的3D表示方法，3D高斯濺射引起了人們的關注，因其快速渲染速度和高質量渲染而聞名。然而，這也伴隨著高內存消耗，例如，一個訓練良好的高斯場可能使用三百萬個高斯基元和超過700 MB 的內存。我們將這種高內存佔用歸因於對基元之間關係的缺乏考慮。在本文中，我們提出了一種記憶效率高的高斯場，名為SUNDAE，採用了光譜修剪和神經補償。一方面，我們在高斯基元集合上構建圖形以建模它們之間的關係，並設計了一個光譜下採樣模塊，以剪除基元同時保留所需信號。另一方面，為了補償剪除高斯的質量損失，我們利用輕量級神經網絡頭來混合濺射特徵，這有效地補償了質量損失，同時在其權重中捕捉基元之間的關係。我們通過廣泛的結果展示了SUNDAE的性能。例如，在Mip-NeRF360數據集上，SUNDAE在145 FPS 時可以實現26.80 PSNR，使用104 MB 內存，而原始的高斯濺射算法在160 FPS 時使用523 MB 內存實現25.60 PSNR。代碼可在https://runyiyang.github.io/projects/SUNDAE/ 公開獲取。

English

Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered attention for its fast rendering speed and high rendering quality. However, this comes with high memory consumption, e.g., a well-trained Gaussian field may utilize three million Gaussian primitives and over 700 MB of memory. We credit this high memory footprint to the lack of consideration for the relationship between primitives. In this paper, we propose a memory-efficient Gaussian field named SUNDAE with spectral pruning and neural compensation. On one hand, we construct a graph on the set of Gaussian primitives to model their relationship and design a spectral down-sampling module to prune out primitives while preserving desired signals. On the other hand, to compensate for the quality loss of pruning Gaussians, we exploit a lightweight neural network head to mix splatted features, which effectively compensates for quality losses while capturing the relationship between primitives in its weights. We demonstrate the performance of SUNDAE with extensive results. For example, SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB memory, on the Mip-NeRF360 dataset. Codes are publicly available at https://runyiyang.github.io/projects/SUNDAE/.