Style-NeRF2NeRF：從風格對齊的多視角影像進行的3D風格轉移

Style-NeRF2NeRF: 3D Style Transfer From Style-Aligned Multi-View Images

June 19, 2024

作者: Haruo Fujiwara, Yusuke Mukuta, Tatsuya Harada

cs.AI

摘要

我們提出了一個簡單而有效的流程，用於為風格化3D場景，利用2D圖像擴散模型的能力。給定從一組多視角圖像重建的NeRF模型，我們通過使用由風格對齊的圖像到圖像擴散模型生成的風格化圖像來優化源NeRF模型，從而執行3D風格轉移。給定目標風格提示，我們首先通過利用具有共享注意機制的深度條件擴散模型生成感知上相似的多視角圖像。接下來，基於風格化的多視角圖像，我們提出使用從預先訓練的CNN模型提取的特徵圖來基於切片Wasserstein損失來引導風格轉移過程。我們的流程包括解耦的步驟，使用戶可以測試各種提示想法並在進入NeRF微調階段之前預覽風格化的3D結果。我們展示了我們的方法可以將多種藝術風格轉移到現實世界的3D場景，並具有競爭力的質量。

English

We propose a simple yet effective pipeline for stylizing a 3D scene, harnessing the power of 2D image diffusion models. Given a NeRF model reconstructed from a set of multi-view images, we perform 3D style transfer by refining the source NeRF model using stylized images generated by a style-aligned image-to-image diffusion model. Given a target style prompt, we first generate perceptually similar multi-view images by leveraging a depth-conditioned diffusion model with an attention-sharing mechanism. Next, based on the stylized multi-view images, we propose to guide the style transfer process with the sliced Wasserstein loss based on the feature maps extracted from a pre-trained CNN model. Our pipeline consists of decoupled steps, allowing users to test various prompt ideas and preview the stylized 3D result before proceeding to the NeRF fine-tuning stage. We demonstrate that our method can transfer diverse artistic styles to real-world 3D scenes with competitive quality.