GaussianSR：具有2D扩散先验的3D高斯超分辨率

GaussianSR: 3D Gaussian Super-Resolution with 2D Diffusion Priors

June 14, 2024

作者: Xiqian Yu, Hanxin Zhu, Tianyu He, Zhibo Chen

cs.AI

摘要

从低分辨率输入视图实现高分辨率新视角合成（HRNVS）是一项具有挑战性的任务，因为缺乏高分辨率数据。先前的方法优化了从低分辨率输入视图到高分辨率神经辐射场（NeRF），但渲染速度较慢。在这项工作中，我们基于3D高斯飞溅（3DGS）的方法，因为它能够以更快的渲染速度生成高质量图像。为了缓解用于更高分辨率合成的数据短缺问题，我们提出利用现成的2D扩散先验，通过得分蒸馏采样（SDS）将2D知识提炼到3D中。然而，将SDS直接应用于基于高斯的3D超分辨率会导致不受欢迎和冗余的3D高斯基元，这是由生成先验带来的随机性所致。为了减轻这个问题，我们引入了两种简单而有效的技术来减少SDS引入的随机干扰。具体来说，我们1）通过一个退火策略缩小SDS中扩散时间步的范围；2）在致密化过程中随机丢弃冗余的高斯基元。大量实验证明，我们提出的GaussainSR能够在合成和真实数据集上仅通过低分辨率输入实现HRNVS的高质量结果。项目页面：https://chchnii.github.io/GaussianSR/

English

Achieving high-resolution novel view synthesis (HRNVS) from low-resolution input views is a challenging task due to the lack of high-resolution data. Previous methods optimize high-resolution Neural Radiance Field (NeRF) from low-resolution input views but suffer from slow rendering speed. In this work, we base our method on 3D Gaussian Splatting (3DGS) due to its capability of producing high-quality images at a faster rendering speed. To alleviate the shortage of data for higher-resolution synthesis, we propose to leverage off-the-shelf 2D diffusion priors by distilling the 2D knowledge into 3D with Score Distillation Sampling (SDS). Nevertheless, applying SDS directly to Gaussian-based 3D super-resolution leads to undesirable and redundant 3D Gaussian primitives, due to the randomness brought by generative priors. To mitigate this issue, we introduce two simple yet effective techniques to reduce stochastic disturbances introduced by SDS. Specifically, we 1) shrink the range of diffusion timestep in SDS with an annealing strategy; 2) randomly discard redundant Gaussian primitives during densification. Extensive experiments have demonstrated that our proposed GaussainSR can attain high-quality results for HRNVS with only low-resolution inputs on both synthetic and real-world datasets. Project page: https://chchnii.github.io/GaussianSR/