2024年AIM稀疏神经渲染挑战赛：数据集与基准测试

AIM 2024 Sparse Neural Rendering Challenge: Dataset and Benchmark

September 23, 2024

作者: Michal Nazarczuk, Thomas Tanay, Sibi Catley-Chandar, Richard Shaw, Radu Timofte, Eduardo Pérez-Pellitero

cs.AI

摘要

最近，可微分渲染和神经渲染的发展在各种2D和3D任务中取得了令人瞩目的突破，例如新视角合成、3D重建。通常，可微分渲染依赖于对场景进行密集视点覆盖，以便仅通过外观观察就能将几何形状与之区分开来。当只有少量输入视图可用时，会出现一些挑战，通常被称为稀疏或少样本神经渲染。由于这是一个欠约束的问题，大多数现有方法引入了正则化的使用，以及各种学习和手工制作的先验。稀疏渲染文献中一个经常出现的问题是缺乏一个统一的、最新的数据集和评估协议。在密集重建文献中，高分辨率数据集是标准的，而稀疏渲染方法通常使用低分辨率图像进行评估。此外，数据拆分在不同文献中不一致，测试的地面真实图像通常是公开的，这可能导致过拟合。在这项工作中，我们提出了稀疏渲染（SpaRe）数据集和基准。我们引入了一个新的数据集，遵循DTU MVS数据集的设置。该数据集由基于合成高质量资产的97个新场景组成。每个场景最多具有64个相机视图和7种光照配置，分辨率为1600x1200。我们发布了82个场景的训练拆分，以促进通用方法，并为验证和测试集提供了一个在线评估平台，其中地面真实图像保持隐藏。我们提出了两种不同的稀疏配置（分别为3和9个输入图像）。这为可重现的评估提供了一个强大且便捷的工具，使研究人员能够轻松访问具有最先进性能得分的公共排行榜。网址：https://sparebenchmark.github.io/

English

Recent developments in differentiable and neural rendering have made impressive breakthroughs in a variety of 2D and 3D tasks, e.g. novel view synthesis, 3D reconstruction. Typically, differentiable rendering relies on a dense viewpoint coverage of the scene, such that the geometry can be disambiguated from appearance observations alone. Several challenges arise when only a few input views are available, often referred to as sparse or few-shot neural rendering. As this is an underconstrained problem, most existing approaches introduce the use of regularisation, together with a diversity of learnt and hand-crafted priors. A recurring problem in sparse rendering literature is the lack of an homogeneous, up-to-date, dataset and evaluation protocol. While high-resolution datasets are standard in dense reconstruction literature, sparse rendering methods often evaluate with low-resolution images. Additionally, data splits are inconsistent across different manuscripts, and testing ground-truth images are often publicly available, which may lead to over-fitting. In this work, we propose the Sparse Rendering (SpaRe) dataset and benchmark. We introduce a new dataset that follows the setup of the DTU MVS dataset. The dataset is composed of 97 new scenes based on synthetic, high-quality assets. Each scene has up to 64 camera views and 7 lighting configurations, rendered at 1600x1200 resolution. We release a training split of 82 scenes to foster generalizable approaches, and provide an online evaluation platform for the validation and test sets, whose ground-truth images remain hidden. We propose two different sparse configurations (3 and 9 input images respectively). This provides a powerful and convenient tool for reproducible evaluation, and enable researchers easy access to a public leaderboard with the state-of-the-art performance scores. Available at: https://sparebenchmark.github.io/