VisualPatchWorld: 作為規劃之潛在結構表徵的碼世界模型

摘要

不同的研究線路對「世界模型」一詞各有界定，但它們的共同目標是：以支援感知、模擬與規劃的形式，捕捉世界在行動作用下的演化過程。兩種顯著的實現方式分別為：在連續向量空間中學習動力學的神經預測器，以及暴露明確狀態與物理定律的手工建構物理引擎。神經預測器可從數據中規模化學習，但動力學形式隱含其中；物理引擎則可檢視與編輯，卻難以大規模建構。我們提出 VisualPatchWorld (VPW)，該方法將世界動力學表示為程式碼。VPW 首先透過短暫主動探測選取定性動力學形式，再從記錄的狀態-行動軌跡中，透過最小化多步預測誤差來擬合該形式的自由參數。所產生的程式可像模擬器一樣向前滾動運算，可檢視原始碼形式，並可用於模型預測控制；影像衍生的場景圖可在重新規劃時提供即時狀態。與先前的基於程式碼的世界模型相比，VPW 平均規劃成功率達 69.0%，並超越最強程式碼基線 23.5 個百分點。最大的提升出現在正確選取定性動力學至關重要的情境中。在同一規劃器下，所誘導的模型在導航與抓取豐富的控制任務上接近真實引擎的成功率；在接觸豐富的推動物件任務中仍存在殘餘差距，而在引擎中檢查少數有前景的規劃方案即可縮小大部分差距。這些結果為自動建構可用於規劃的程式碼世界模型開闢了實用途徑。程式碼見 https://github.com/HKBU-KnowComp/VisualPatchWorld/。

English

Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at https://github.com/HKBU-KnowComp/VisualPatchWorld/.