動態神經光線場模型應用於足球場景

摘要

長期存在的新視角合成問題在許多應用中具有重要意義，尤其在體育廣播領域。對於足球比賽動作的照片逼真新視角合成對廣播行業具有極大的吸引力。然而，目前只有少數工業解決方案被提出，甚至更少能夠實現合成重播的接近廣播質量。除了在比賽場地周圍設置多個靜態攝影機外，最佳專有系統幾乎沒有披露有關其內部運作的任何信息。利用多個靜態攝影機來完成這樣的任務確實在文獻中很少被探討，主要是因為缺乏公開數據集：重建大規模、主要靜態環境，其中包含小型、快速移動的元素。最近，神經輻射場的出現在許多新視角合成應用中帶來了驚人的進展，利用深度學習原則在最具挑戰性的環境中產生照片逼真的結果。在這項工作中，我們探討了基於動態神經輻射場（即用於重建一般動態內容的神經模型）的解決方案是否可行。我們構建了合成足球環境並對其進行多次實驗，識別有助於使用動態神經輻射場重建足球場景的關鍵組件。我們表明，儘管這種方法無法完全滿足目標應用的質量要求，但它提出了通往成本效益高、自動化解決方案的有希望途徑。我們還公開提供了我們的工作數據集和代碼，旨在鼓勵研究社區進一步努力從事動態足球場景的新視角合成任務。有關代碼、數據和視頻結果，請參見https://soccernerfs.isach.be。

English

The long-standing problem of novel view synthesis has many applications, notably in sports broadcasting. Photorealistic novel view synthesis of soccer actions, in particular, is of enormous interest to the broadcast industry. Yet only a few industrial solutions have been proposed, and even fewer that achieve near-broadcast quality of the synthetic replays. Except for their setup of multiple static cameras around the playfield, the best proprietary systems disclose close to no information about their inner workings. Leveraging multiple static cameras for such a task indeed presents a challenge rarely tackled in the literature, for a lack of public datasets: the reconstruction of a large-scale, mostly static environment, with small, fast-moving elements. Recently, the emergence of neural radiance fields has induced stunning progress in many novel view synthesis applications, leveraging deep learning principles to produce photorealistic results in the most challenging settings. In this work, we investigate the feasibility of basing a solution to the task on dynamic NeRFs, i.e., neural models purposed to reconstruct general dynamic content. We compose synthetic soccer environments and conduct multiple experiments using them, identifying key components that help reconstruct soccer scenes with dynamic NeRFs. We show that, although this approach cannot fully meet the quality requirements for the target application, it suggests promising avenues toward a cost-efficient, automatic solution. We also make our work dataset and code publicly available, with the goal to encourage further efforts from the research community on the task of novel view synthesis for dynamic soccer scenes. For code, data, and video results, please see https://soccernerfs.isach.be.

動態神經光線場模型應用於足球場景

Dynamic NeRFs for Soccer Scenes

摘要

Support