一叶知秋

摘要

计算机视觉领域的一个长期目标是实现视频中的运动建模，然而运动背后的表征——即导致物体形变与移动的不可见物理相互作用——至今仍少有研究。本文致力于探索如何从视觉观测中还原不可见的作用力，例如通过观察树叶飘落的过程来估算风场。我们的核心创新在于提出了一种端到端的可微分逆向图形框架，能够直接从视频数据中联合建模物体几何、物理属性及相互作用。通过反向传播算法，该方法实现了从物体运动中还原力场表征的能力。我们在合成场景与真实场景中验证了本方法，结果表明其能够从视频中推断出合理的力场分布。此外，我们还展示了该方法在物理启发的视频生成与编辑等领域的应用潜力。我们期望该研究能为理解像素背后的物理过程、弥合视觉与物理之间的鸿沟提供新的思路。更多视频结果请访问我们的{项目页面}https://chaoren2357.github.io/seeingthewind/。

English

A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our https://chaoren2357.github.io/seeingthewind/{project page}.

Seeing the Wind from a Falling Leaf

摘要

Support