将面部交互图网络扩展到现实世界场景

摘要

准确模拟现实世界物体动态对于各种应用至关重要，如机器人技术、工程、图形学和设计。为了更好地捕捉诸如接触和摩擦等复杂真实动态，基于图网络的学习模拟器最近显示出巨大潜力。然而，将这些学习模拟器应用于真实场景面临两个主要挑战：首先，将学习模拟器扩展到处理真实世界场景的复杂性，这可能涉及数百个具有复杂3D形状的物体；其次，处理来自感知而非3D状态信息的输入。在这里，我们介绍了一种方法，大幅减少了运行基于图的学习模拟器所需的内存。基于这种内存高效的模拟模型，我们随后提出了一个感知界面，采用可编辑的 NeRFs 形式，可以将真实世界场景转换为结构化表示，以便图网络模拟器进行处理。我们展示了我们的方法使用的内存明显少于先前基于图的模拟器，同时保持其准确性，并且在合成环境中学习的模拟器可以应用于从多个摄像机角度捕获的真实世界场景。这为将学习模拟器的应用扩展到仅在推断时可用感知信息的设置铺平了道路。

English

Accurately simulating real world object dynamics is essential for various applications such as robotics, engineering, graphics, and design. To better capture complex real dynamics such as contact and friction, learned simulators based on graph networks have recently shown great promise. However, applying these learned simulators to real scenes comes with two major challenges: first, scaling learned simulators to handle the complexity of real world scenes which can involve hundreds of objects each with complicated 3D shapes, and second, handling inputs from perception rather than 3D state information. Here we introduce a method which substantially reduces the memory required to run graph-based learned simulators. Based on this memory-efficient simulation model, we then present a perceptual interface in the form of editable NeRFs which can convert real-world scenes into a structured representation that can be processed by graph network simulator. We show that our method uses substantially less memory than previous graph-based simulators while retaining their accuracy, and that the simulators learned in synthetic environments can be applied to real world scenes captured from multiple camera angles. This paves the way for expanding the application of learned simulators to settings where only perceptual information is available at inference time.

将面部交互图网络扩展到现实世界场景

Scaling Face Interaction Graph Networks to Real World Scenes

摘要

Support