学习世界如何演化：通过潜在动态推理实现外推式视频世界模型

摘要

世界按照其动力学，即运动规律，持续演化。然而，主流视频扩散模型主要拟合像素，而不对像素随时间的变化过程进行建模。因此，它们能生成视觉上合理的帧，但可能无法精确遵循运动规律。为了纯从像素中捕捉动力学，我们提出了潜变量动力学推理（Latent Dynamics Reasoning, LDR）。LDR将潜变量转移建模为显式的运动学积分，其中低阶动力学通过数值积分求解，模型仅回归驱动推演的第三阶及更高阶残差。为使该积分具有更好的外推能力，LDR在结构化潜表示上执行积分，而非密集卷积特征。参照PhyWorld，我们在一个受控的白盒物理基准上验证LDR，该基准涵盖五项任务（匀速运动、抛物线、碰撞、弹跳、逼近），重点关注能揭示模型是否真正学习了底层动力学的分布外场景。LDR对外推所学动力学的表现远优：在256²分辨率下，无论是单任务训练还是多任务联合训练，其分布内与分布外误差之间的差距比视频扩散基线小20倍以上，同时参数量减少26倍，运行速度快143倍。LDR甚至能在剧烈分布偏移下泛化：例如，仅在红球从左向右运动的数据上训练后，它能正确预测蓝方块从右向左的运动。据我们所知，这是首个能够将所学动力学外推至训练分布之外的视频世界模型。项目主页：https://lat-dyn-reason.github.io/

English

The world evolves following its dynamics, i.e., its laws of motion. However, leading video diffusion models largely fit the pixels without modeling how the pixels transit over time. Thus, they render visually plausible frames but may not accurately obey the laws. To capture the dynamics purely from pixels, we introduce Latent Dynamics Reasoning (LDR). LDR casts the latent transition as an explicit kinematic integration, where the lower-order dynamics are integrated numerically and the model regresses only the third- and higher-order residual that drives the rollout. For this integration to extrapolate better, LDR runs it on a structured latent rather than dense convolutional features. Following PhyWorld, we validate LDR on a controlled white-box physics benchmark spanning five tasks (uniform motion, parabola, collision, bouncing, looming), focusing on out-of-distribution scenarios that reveal whether a model has truly learned the underlying dynamics. LDR extrapolates the learned dynamics far better: the gap between its in- and out-of-distribution error is over 20times smaller than the video diffusion baseline's, under both single- and joint-task training at 256^2 resolution, while using 26times fewer parameters and running 143times faster. LDR can even generalize under severe shift: for example, trained only on red balls moving left-to-right, it correctly predicts the motion of a blue square moving right-to-left. To our knowledge, this is the first video world model that extrapolates learned dynamics beyond its training distribution. Project page: https://lat-dyn-reason.github.io/