PhysTwin：基于物理约束的视频可变形物体重建与仿真

PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos

March 23, 2025

作者: Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, Yunzhu Li

cs.AI

摘要

创建现实世界物体的物理数字孪生体在机器人技术、内容创作和扩展现实（XR）领域具有巨大潜力。本文介绍了一种名为PhysTwin的创新框架，该框架利用交互中动态物体的稀疏视频，生成照片级真实感且物理逼真的实时交互虚拟复制品。我们的方法围绕两个核心组件展开：（1）一种物理信息表示法，结合了弹簧-质量模型以实现逼真的物理模拟，生成式形状模型用于几何构建，以及高斯样条用于渲染；（2）一种新颖的多阶段、基于优化的逆向建模框架，能够从视频中重建完整几何结构，推断密集物理属性，并复制真实外观。我们的方法将逆向物理框架与视觉感知线索相结合，即使在部分遮挡和视角受限的情况下，也能实现高保真重建。PhysTwin支持建模多种可变形物体，包括绳索、毛绒玩具、布料和快递包裹。实验表明，PhysTwin在重建、渲染、未来预测及新交互下的模拟方面均优于竞争方法。我们进一步展示了其在交互式实时仿真和基于模型的机器人运动规划中的应用。

English

Creating a physical digital twin of a real-world object has immense potential in robotics, content creation, and XR. In this paper, we present PhysTwin, a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive virtual replica. Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, generative shape models for geometry, and Gaussian splats for rendering; and (2) a novel multi-stage, optimization-based inverse modeling framework that reconstructs complete geometry, infers dense physical properties, and replicates realistic appearance from videos. Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints. PhysTwin supports modeling various deformable objects, including ropes, stuffed animals, cloth, and delivery packages. Experiments show that PhysTwin outperforms competing methods in reconstruction, rendering, future prediction, and simulation under novel interactions. We further demonstrate its applications in interactive real-time simulation and model-based robotic motion planning.