WorldCraft：從攝影機導航到物體操控的互動式影片世界模型

摘要

近期基于视频的世界模型已使像素级别的环境具备了相机层面的交互能力：用户可以自由调整视角，同时模型能够生成连贯的视觉延续。然而，这些模型的“动作空间”仍不完整：用户只能移动相机，却无法对单个物体施加操作。由于真实世界的交互本质上是“以物体为中心”的，这类模型更像是被动的场景观察者，而非真正可操控的环境。为此，我们提出 **WorldCraft** 框架，将交互式视频世界模型的范畴从相机导航扩展至物体级轨迹动作。用户只需点击并绘制一条路径，WorldCraft 即可生成未来帧，其中被选中的物体沿指定轨迹移动，同时相机继续在场景中自由导航。WorldCraft 通过一套以轨迹为核心的操控流水线实现这一功能：首先，**归一化世界轨迹（NWT）** 将用户绘制的运动表示在相机不变的全局世界坐标系中，并根据当前相机姿态动态重新投影，从而将物体运动与相机引起的屏幕空间位移相分离；接着，**空间路径 LoRA（SP-LoRA）** 将这一世界空间信号注入模型的空间操控通路，在保留预训练相机控制器的同时增添物体操控能力；最后，**轨迹锚定状态持久化（TASP）** 将世界轨迹视为持久的空间状态，并在基于轨迹条件生成后刷新自回归记忆，确保被移动的物体在离开相机视野后重新出现时仍位于更新后的位置。实验表明，WorldCraft 实现了精确的物体控制，在仅进行相机控制的评估中保持了基于视频的世界模型的相机保真度，并能跨越包含镜头外移动的长程自回归生成过程维持物体状态。

English

Recent video-based world models have made pixel-space environments interactive at the camera level: users can navigate viewpoints while the model generates coherent visual continuations. Yet their action spaces remain incomplete: users can move the camera, but cannot act on individual objects. Since real-world interaction is inherently object-centric, such models remain closer to passive scene observers than truly manipulable environments. We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions. Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory while the camera continues to navigate the scene. WorldCraft achieves this through a trajectory-centric control pipeline: First, Normalized World Trajectory (NWT) represents user-drawn motion in a camera-invariant world coordinate system and dynamically re-projects it under the current camera pose, separating object motion from camera-induced screen-space displacement; Spatial-Pathway LoRA (SP-LoRA) then injects this world-space signal through the model's spatial-control pathway, adding object manipulation capability while preserving the pretrained camera controller; finally, Trajectory-Anchored State Persistence (TASP) treats the world trajectory as a persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation, allowing moved objects to reappear at their updated positions after leaving the camera view. Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts with off-camera excursions.