具有神经补偿的频谱修剪高斯场

Spectrally Pruned Gaussian Fields with Neural Compensation

May 1, 2024

作者: Runyi Yang, Zhenxin Zhu, Zhou Jiang, Baijun Ye, Xiaoxue Chen, Yifei Zhang, Yuantao Chen, Jian Zhao, Hao Zhao

cs.AI

摘要

最近，作为一种新颖的3D表示方法，3D高斯飞溅引起了人们的关注，因其快速渲染速度和高渲染质量。然而，这也带来了高内存消耗，例如，一个经过良好训练的高斯场可能利用三百万个高斯基元和超过700 MB的内存。我们将这种高内存占用归因于对基元之间关系缺乏考虑。在本文中，我们提出了一种名为SUNDAE的内存高效的高斯场，采用谱修剪和神经补偿。一方面，我们在高斯基元集上构建图来建模它们的关系，并设计了一个谱下采样模块，以剪除基元同时保留所需信号。另一方面，为了补偿修剪高斯带来的质量损失，我们利用一个轻量级神经网络头来混合飞溅特征，有效地补偿了质量损失，同时在其权重中捕捉基元之间的关系。我们通过广泛的结果展示了SUNDAE的性能。例如，在Mip-NeRF360数据集上，SUNDAE在145 FPS时可以实现26.80的PSNR，使用104 MB内存，而原始高斯飞溅算法在160 FPS时使用523 MB内存，实现25.60的PSNR。代码可在https://runyiyang.github.io/projects/SUNDAE/公开获取。

English

Recently, 3D Gaussian Splatting, as a novel 3D representation, has garnered attention for its fast rendering speed and high rendering quality. However, this comes with high memory consumption, e.g., a well-trained Gaussian field may utilize three million Gaussian primitives and over 700 MB of memory. We credit this high memory footprint to the lack of consideration for the relationship between primitives. In this paper, we propose a memory-efficient Gaussian field named SUNDAE with spectral pruning and neural compensation. On one hand, we construct a graph on the set of Gaussian primitives to model their relationship and design a spectral down-sampling module to prune out primitives while preserving desired signals. On the other hand, to compensate for the quality loss of pruning Gaussians, we exploit a lightweight neural network head to mix splatted features, which effectively compensates for quality losses while capturing the relationship between primitives in its weights. We demonstrate the performance of SUNDAE with extensive results. For example, SUNDAE can achieve 26.80 PSNR at 145 FPS using 104 MB memory while the vanilla Gaussian splatting algorithm achieves 25.60 PSNR at 160 FPS using 523 MB memory, on the Mip-NeRF360 dataset. Codes are publicly available at https://runyiyang.github.io/projects/SUNDAE/.