一叶知秋

摘要

计算机视觉领域长期致力于从视频中建模物体运动，然而运动背后的表征——即导致物体形变与移动的不可见物理相互作用——仍属未充分探索的领域。本文研究如何从视觉观测中复原不可见的作用力，例如通过观察落叶飘向来估算风场。我们的核心创新在于提出端到端可微分的逆向图形学框架，该框架能够直接从视频中联合建模物体几何、物理属性与相互作用。通过反向传播算法，我们的方法实现了从物体运动中复原力场表征。我们在合成场景与真实场景中验证了该方法，结果表明其能从视频中推断出合理的力场分布。此外，我们还展示了该方法在基于物理的视频生成与编辑等领域的应用潜力。我们期望该研究能为理解像素背后的物理过程、弥合视觉与物理之间的鸿沟提供新思路。更多视频结果请访问项目页面：https://chaoren2357.github.io/seeingthewind/

English

A longstanding goal in computer vision is to model motions from videos, while the representations behind motions, i.e. the invisible physical interactions that cause objects to deform and move, remain largely unexplored. In this paper, we study how to recover the invisible forces from visual observations, e.g., estimating the wind field by observing a leaf falling to the ground. Our key innovation is an end-to-end differentiable inverse graphics framework, which jointly models object geometry, physical properties, and interactions directly from videos. Through backpropagation, our approach enables the recovery of force representations from object motions. We validate our method on both synthetic and real-world scenarios, and the results demonstrate its ability to infer plausible force fields from videos. Furthermore, we show the potential applications of our approach, including physics-based video generation and editing. We hope our approach sheds light on understanding and modeling the physical process behind pixels, bridging the gap between vision and physics. Please check more video results in our https://chaoren2357.github.io/seeingthewind/{project page}.

Seeing the Wind from a Falling Leaf

摘要

Support