面向機器人控制的上下文世界建模

摘要

現代的視覺-語言-動作（VLA）模型常常難以泛化到新的配置環境中，例如改變的攝影機視角或機器人構型，這是因為這些模型通常僅以當前的觀察與語言指令為條件。這些模型忽略了底層系統配置這個變數，隱含地假設訓練時遇到的固定執行環境，導致任何新環境都需要耗費大量數據進行微調。在這項工作中，我們提出了「情境內世界模型」（ICWM），這是一個將系統辨識視為情境內適應問題的框架。ICWM 使機器人策略能夠從一小段自我生成、與任務無關的互動歷史中自主推斷出必要的系統變數。與傳統的情境內學習（利用示範來指定要執行的任務）不同，ICWM 是利用情境視窗來理解系統的運作方式。透過在任務執行前處理這些互動，模型能隱含地捕捉當前系統的世界動態，從而無需更新參數即可適應新的配置。在模擬環境與真實機器人平台上進行的廣泛實驗證明，ICWM 在應對新攝影機視角時，顯著優於標準的 VLA 基準模型。

English

Modern Vision-Language-Action (VLA) models often fail to generalize to novel setups, such as altered camera viewpoints or robot morphologies, because they are typically conditioned only on current observations and language instructions. By ignoring the underlying system configuration as a variable, these models implicitly assume a fixed execution context encountered during training, necessitating data-intensive fine-tuning for any new environment. In this work, we introduce In-Context World Modeling (ICWM), a framework that treats system identification as an in-context adaptation problem. ICWM enables robot policies to autonomously infer essential system variables from a short history of self-generated, task-agnostic interactions. Unlike traditional In-Context Learning that uses demonstrations to specify what task to perform, ICWM leverages the context window to understand how the system operates. By processing these interactions before task execution, the model implicitly captures the world dynamics of the current system, enabling adaptation to novel configurations without parameter updates. Extensive experiments in simulation and on real-world robot platforms demonstrate that ICWM significantly outperforms standard VLA baselines on novel camera viewpoints.