PhysDreamer：基于物理的视频生成与3D对象的交互

PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation

April 19, 2024

作者: Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Y. Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, William T. Freeman

cs.AI

摘要

创造沉浸式虚拟体验时，逼真的物体互动至关重要，然而合成对新颖互动做出逼真3D物体动态响应仍然是一个重大挑战。与无条件或文本条件动态生成不同，动作条件动态需要感知物体的物理材料属性，并将3D运动预测基于这些属性，如物体的刚度。然而，由于缺乏物质的真实数据，估计物理材料属性是一个未解之谜，因为为真实物体测量这些属性非常困难。我们提出了PhysDreamer，这是一种基于物理的方法，通过利用视频生成模型学习的物体动态先验，赋予静态3D物体交互动态。通过提炼这些先验，PhysDreamer能够合成对新颖互动的逼真物体响应，如外部力或代理操纵。我们在弹性物体的多个示例上展示了我们的方法，并通过用户研究评估了合成互动的逼真程度。PhysDreamer通过使静态3D物体以物理上合理的方式动态响应互动刺激，迈出了实现更具吸引力和逼真的虚拟体验的一步。请访问我们的项目页面：https://physdreamer.github.io/。

English

Realistic object interactions are crucial for creating immersive virtual experiences, yet synthesizing realistic 3D object dynamics in response to novel interactions remains a significant challenge. Unlike unconditional or text-conditioned dynamics generation, action-conditioned dynamics requires perceiving the physical material properties of objects and grounding the 3D motion prediction on these properties, such as object stiffness. However, estimating physical material properties is an open problem due to the lack of material ground-truth data, as measuring these properties for real objects is highly difficult. We present PhysDreamer, a physics-based approach that endows static 3D objects with interactive dynamics by leveraging the object dynamics priors learned by video generation models. By distilling these priors, PhysDreamer enables the synthesis of realistic object responses to novel interactions, such as external forces or agent manipulations. We demonstrate our approach on diverse examples of elastic objects and evaluate the realism of the synthesized interactions through a user study. PhysDreamer takes a step towards more engaging and realistic virtual experiences by enabling static 3D objects to dynamically respond to interactive stimuli in a physically plausible manner. See our project page at https://physdreamer.github.io/.