VisualPatchWorld: 作为规划潜在结构化表示的代码世界模型

摘要

不同的研究路线以不同的方式使用“世界模型”这一术语，但它们共享一个共同目标：以支持感知、模拟和规划的形式，捕捉世界在行动作用下的演化过程。两种突出的实现方式是：在连续向量空间中学习动力学的神经预测器，以及暴露显式状态和物理定律的手工构建物理引擎。神经预测器可从数据中扩展，但将动力学形式隐含其中；物理引擎则可检查与编辑，但难以大规模构建。我们提出VisualPatchWorld（VPW），它将世界动力学表示为代码。VPW首先通过短暂的主动探测选择一种定性动力学形式，然后通过最小化多步预测误差，从记录的状态-动作轨迹中拟合该形式的自由参数。由此生成的程序可像模拟器一样向前推进，以源代码形式进行检查，并用于模型预测控制；图像衍生的场景图可在重新规划时提供实时状态。在与先前基于代码的世界模型的比较中，VPW实现了69.0%的平均规划成功率，比最强的代码基线高出23.5个百分点。最大的提升出现在正确选择定性动力学至关重要的情况下。在相同规划器下，诱导模型在导航和富含抓取的控制任务中接近真实物理引擎的成功率；在涉及接触的推挤任务中仍存在残余差距，而在引擎中检查少数有前景的方案即可弥补大部分差距。这些结果建立了一条实用路径，可自动构建对规划有用的代码世界模型。代码已开源：https://github.com/HKBU-KnowComp/VisualPatchWorld/。

English

Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.