PhysTwin：基於物理約束的變形物體視頻重建與模擬

PhysTwin: Physics-Informed Reconstruction and Simulation of Deformable Objects from Videos

March 23, 2025

作者: Hanxiao Jiang, Hao-Yu Hsu, Kaifeng Zhang, Hsin-Ni Yu, Shenlong Wang, Yunzhu Li

cs.AI

摘要

創建真實世界物體的物理數位孿生體在機器人技術、內容創作和擴展現實（XR）領域具有巨大潛力。本文介紹了PhysTwin，這是一種新穎的框架，利用互動中動態物體的稀疏視頻來生成照片級逼真且物理上真實、可實時互動的虛擬複製品。我們的方法圍繞兩個關鍵組件展開：(1) 一種物理信息表示法，結合了用於真實物理模擬的彈簧-質量模型、用於幾何的生成形狀模型以及用於渲染的高斯散點；(2) 一種新穎的多階段、基於優化的逆向建模框架，該框架從視頻中重建完整幾何、推斷密集物理屬性並複製逼真外觀。我們的方法將逆向物理框架與視覺感知線索相結合，即使在部分遮擋和視角有限的情況下也能實現高保真重建。PhysTwin支持建模各種可變形物體，包括繩索、填充玩具、布料和快遞包裹。實驗表明，PhysTwin在重建、渲染、未來預測以及新穎互動下的模擬方面優於競爭方法。我們進一步展示了其在互動實時模擬和基於模型的機器人運動規劃中的應用。

English

Creating a physical digital twin of a real-world object has immense potential in robotics, content creation, and XR. In this paper, we present PhysTwin, a novel framework that uses sparse videos of dynamic objects under interaction to produce a photo- and physically realistic, real-time interactive virtual replica. Our approach centers on two key components: (1) a physics-informed representation that combines spring-mass models for realistic physical simulation, generative shape models for geometry, and Gaussian splats for rendering; and (2) a novel multi-stage, optimization-based inverse modeling framework that reconstructs complete geometry, infers dense physical properties, and replicates realistic appearance from videos. Our method integrates an inverse physics framework with visual perception cues, enabling high-fidelity reconstruction even from partial, occluded, and limited viewpoints. PhysTwin supports modeling various deformable objects, including ropes, stuffed animals, cloth, and delivery packages. Experiments show that PhysTwin outperforms competing methods in reconstruction, rendering, future prediction, and simulation under novel interactions. We further demonstrate its applications in interactive real-time simulation and model-based robotic motion planning.