DNA-Rendering：一個多樣性神經演算法庫，用於高保真度的以人為中心渲染。

DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Rendering

July 19, 2023

作者: Wei Cheng, Ruixiang Chen, Wanqi Yin, Siming Fan, Keyu Chen, Honglin He, Huiwen Luo, Zhongang Cai, Jingbo Wang, Yang Gao, Zhengming Yu, Zhengyu Lin, Daxuan Ren, Lei Yang, Ziwei Liu, Chen Change Loy, Chen Qian, Wayne Wu, Dahua Lin, Bo Dai, Kwan-Yee Lin

cs.AI

摘要

真實的以人為中心渲染在計算機視覺和計算機圖形學中扮演著關鍵角色。多年來在算法方面取得了快速進展，然而現有的以人為中心渲染數據集和基準測試在多樣性方面相對貧乏，而這對於渲染效果至關重要。研究人員通常受限於在當前數據集上探索和評估少量渲染問題，而現實應用需要方法能夠在不同場景下保持穩健性。在這項工作中，我們提出了DNA-Rendering，這是一個大規模、高保真度的神經演員渲染人類表現數據庫。DNA-Rendering 具有幾個吸引人的特點。首先，我們的數據集包含超過1500名人類受試者、5000個運動序列和6750萬幀的數據量。其次，我們為每個受試者提供豐富的資產 - 2D/3D 人體關鍵點、前景遮罩、SMPLX 模型、服裝/配飾材料、多視圖圖像和視頻。這些資產提高了當前方法在下游渲染任務上的準確性。第三，我們構建了一個專業的多視圖系統來捕獲數據，其中包含60個同步相機，最大解析度為4096 x 3000，速度為每秒15幀，以及嚴格的相機校準步驟，確保了高質量的資源用於任務訓練和評估。除了數據集，我們提供了一個全面的大規模和定量基準測試，其中包含多個任務，用於評估新型視圖合成、新型姿勢動畫合成和新型身份渲染方法的現有進展。在這篇論文中，我們描述了我們的DNA-Rendering工作，揭示了新的觀察、挑戰和未來發展方向，針對以人為中心的渲染。數據集、代碼和基準測試將在 https://dna-rendering.github.io/ 上公開提供。

English

Realistic human-centric rendering plays a key role in both computer vision and computer graphics. Rapid progress has been made in the algorithm aspect over the years, yet existing human-centric rendering datasets and benchmarks are rather impoverished in terms of diversity, which are crucial for rendering effect. Researchers are usually constrained to explore and evaluate a small set of rendering problems on current datasets, while real-world applications require methods to be robust across different scenarios. In this work, we present DNA-Rendering, a large-scale, high-fidelity repository of human performance data for neural actor rendering. DNA-Rendering presents several alluring attributes. First, our dataset contains over 1500 human subjects, 5000 motion sequences, and 67.5M frames' data volume. Second, we provide rich assets for each subject -- 2D/3D human body keypoints, foreground masks, SMPLX models, cloth/accessory materials, multi-view images, and videos. These assets boost the current method's accuracy on downstream rendering tasks. Third, we construct a professional multi-view system to capture data, which contains 60 synchronous cameras with max 4096 x 3000 resolution, 15 fps speed, and stern camera calibration steps, ensuring high-quality resources for task training and evaluation. Along with the dataset, we provide a large-scale and quantitative benchmark in full-scale, with multiple tasks to evaluate the existing progress of novel view synthesis, novel pose animation synthesis, and novel identity rendering methods. In this manuscript, we describe our DNA-Rendering effort as a revealing of new observations, challenges, and future directions to human-centric rendering. The dataset, code, and benchmarks will be publicly available at https://dna-rendering.github.io/