足球场景的动态神经辐射场模型

摘要

长期存在的新视角合成问题有许多应用，尤其在体育转播中。特别是对足球比赛动作的逼真新视角合成对广播行业具有巨大的吸引力。然而，目前只有少数几种工业解决方案被提出，甚至更少能够实现合成重播的接近广播质量。除了在比赛场地周围设置多个静态摄像头外，最佳专有系统几乎没有透露任何关于其内部运作的信息。利用多个静态摄像头来完成这样的任务在文献中确实很少被探讨，因为缺乏公共数据集：即在重建大规模、主要静态环境中，存在小型、快速移动元素的挑战。最近，神经辐射场的出现在许多新视角合成应用中取得了惊人的进展，利用深度学习原理在最具挑战性的环境中产生逼真的结果。在这项工作中，我们调查了基于动态神经辐射场（即旨在重建一般动态内容的神经模型）的任务解决方案的可行性。我们构建了合成足球环境，并对其进行多次实验，识别有助于使用动态神经辐射场重建足球场景的关键组件。我们表明，尽管这种方法无法完全满足目标应用的质量要求，但它提出了通向成本效益高、自动化解决方案的有希望途径。我们还公开提供了我们的工作数据集和代码，旨在鼓励研究社区进一步努力，致力于动态足球场景的新视角合成任务。有关代码、数据和视频结果，请访问https://soccernerfs.isach.be。

English

The long-standing problem of novel view synthesis has many applications, notably in sports broadcasting. Photorealistic novel view synthesis of soccer actions, in particular, is of enormous interest to the broadcast industry. Yet only a few industrial solutions have been proposed, and even fewer that achieve near-broadcast quality of the synthetic replays. Except for their setup of multiple static cameras around the playfield, the best proprietary systems disclose close to no information about their inner workings. Leveraging multiple static cameras for such a task indeed presents a challenge rarely tackled in the literature, for a lack of public datasets: the reconstruction of a large-scale, mostly static environment, with small, fast-moving elements. Recently, the emergence of neural radiance fields has induced stunning progress in many novel view synthesis applications, leveraging deep learning principles to produce photorealistic results in the most challenging settings. In this work, we investigate the feasibility of basing a solution to the task on dynamic NeRFs, i.e., neural models purposed to reconstruct general dynamic content. We compose synthetic soccer environments and conduct multiple experiments using them, identifying key components that help reconstruct soccer scenes with dynamic NeRFs. We show that, although this approach cannot fully meet the quality requirements for the target application, it suggests promising avenues toward a cost-efficient, automatic solution. We also make our work dataset and code publicly available, with the goal to encourage further efforts from the research community on the task of novel view synthesis for dynamic soccer scenes. For code, data, and video results, please see https://soccernerfs.isach.be.

足球场景的动态神经辐射场模型

Dynamic NeRFs for Soccer Scenes

摘要

Support