PhysDreamer：基於物理的方式通過影片生成與3D物體互動

PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

April 19, 2024

作者: Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, William T. Freeman

cs.AI

摘要

寫實的物體互動對於創造身臨其境的虛擬體驗至關重要，然而合成對新型互動的寫實3D物體動態仍然是一個重大挑戰。與無條件或文本條件動態生成不同，動作條件動態需要感知物體的物理材料特性，並基於這些特性（如物體的硬度）來預測3D運動。然而，由於缺乏材料真實數據，估算物理材料特性是一個未解決的問題，因為為真實物體測量這些特性非常困難。我們提出PhysDreamer，這是一種基於物理的方法，通過利用視頻生成模型學習的物體動態先驗知識，賦予靜態3D物體互動動態。通過提煉這些先驗知識，PhysDreamer實現了對新型互動（如外部力或代理操作）的寫實物體響應的合成。我們在彈性物體的多個示例上展示了我們的方法，並通過用戶研究評估了合成互動的寫實性。PhysDreamer通過使靜態3D物體能夠以物理合理的方式動態響應互動刺激，邁出了邁向更具吸引力和寫實的虛擬體驗的一步。請查看我們的項目頁面：https://physdreamer.github.io/。

English

Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties for real objects is highly difficult. We present PhysDreamer, a physics-based approach that endows static 3D objects with interactive dynamics by leveraging the object dynamics priors learned by video generation models. By distilling these priors, PhysDreamer enables the synthesis of realistic object responses to novel interactions, such as external forces or agent manipulations. We demonstrate our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study. PhysDreamer takes a step towards more engaging and realistic virtual experiences by enabling static 3D objects to dynamically respond to interactive stimuli in a physically plausible manner. See our project page at https://physdreamer.github.io/.