學習世界如何演變：透過潛在動力學推理實現外推式視頻世界模型

摘要

世界按照其動力學——即運動定律——演進。然而，主流影片擴散模型在很大程度上僅是擬合像素，而並未建模像素隨時間的轉換過程。因此，它們能渲染出視覺上合理的幀，但未必精確遵循物理定律。為純粹從像素中捕捉動力學，我們提出了潛在動力學推理（LDR）。LDR將潛在狀態轉換明確表述為運動學積分，其中低階動力學以數值方式積分，模型僅回歸驅動推演的第三階及以上殘差。為使此積分具備更強的外推能力，LDR在結構化潛在表示而非密集卷積特徵上進行運算。遵循PhyWorld，我們在涵蓋五項任務（勻速運動、拋物線、碰撞、彈跳、逼近）的受控白盒物理基準上驗證LDR，並聚焦於分布外場景，以揭示模型是否真正學習到了底層動力學。LDR能大幅更優地外推所學動力學：在256²解析度下，無論是單任務還是聯合任務訓練，其分布內外誤差之間的差距比影片擴散基線小逾20倍，同時參數量減少26倍，運行速度提升143倍。LDR甚至能在嚴重的分布偏移下進行泛化：例如，僅以紅球自左向右運動的數據訓練後，它能正確預測藍色方塊自右向左的運動。據我們所知，這是首個能將所學動力學外推至訓練分布之外的影片世界模型。專案頁面：https://lat-dyn-reason.github.io/

English

The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20times smaller than the video diffusion baseline's, under both single- and joint-task training at 256^2 resolution, while using 26times fewer parameters and running 143times faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/