TEDRA：基於文本的動態和逼真演員編輯

TEDRA: Text-based Editing of Dynamic and Photoreal Actors

August 28, 2024

作者: Basavaraj Sunagad, Heming Zhu, Mohit Mendiratta, Adam Kortylewski, Christian Theobalt, Marc Habermann

cs.AI

摘要

在過去的幾年中，已取得顯著進展，僅通過真人影片創建逼真且可駕駛的3D化身。然而，一個核心的挑戰是通過文本描述對服裝風格進行精細和用戶友好的編輯。為此，我們提出了TEDRA，這是第一種允許基於文本對化身進行編輯的方法，該方法保持了化身的高保真度、時空一致性以及動態性，並實現了骨架姿勢和視角控制。我們首先訓練一個模型來創建一個可控且高保真度的真實演員的數字副本。接下來，我們通過在從不同攝像機角度捕捉的真實角色的各種幀上對預訓練的生成擴散模型進行微調個性化，確保數字表示忠實地捕捉真實人物的動態和運動。這個兩階段的過程為我們的動態人類化身編輯方法奠定了基礎。利用這個個性化的擴散模型，我們在基於模型的引導框架內使用我們的個性化正常對齊分數蒸餾抽樣（PNA-SDS）來根據提供的文本提示修改動態化身。此外，我們提出了一種時間步驟退火策略，以確保高質量的編輯。我們的結果顯示，在功能性和視覺質量方面，明顯優於先前的工作。

English

Over the past years, significant progress has been made in creating photorealistic and drivable 3D avatars solely from videos of real humans. However, a core remaining challenge is the fine-grained and user-friendly editing of clothing styles by means of textual descriptions. To this end, we present TEDRA, the first method allowing text-based edits of an avatar, which maintains the avatar's high fidelity, space-time coherency, as well as dynamics, and enables skeletal pose and view control. We begin by training a model to create a controllable and high-fidelity digital replica of the real actor. Next, we personalize a pretrained generative diffusion model by fine-tuning it on various frames of the real character captured from different camera angles, ensuring the digital representation faithfully captures the dynamics and movements of the real person. This two-stage process lays the foundation for our approach to dynamic human avatar editing. Utilizing this personalized diffusion model, we modify the dynamic avatar based on a provided text prompt using our Personalized Normal Aligned Score Distillation Sampling (PNA-SDS) within a model-based guidance framework. Additionally, we propose a time step annealing strategy to ensure high-quality edits. Our results demonstrate a clear improvement over prior work in functionality and visual quality.