WorldCraft：交互式视频世界模型中从相机导航到物体操控

摘要

近年来，基于视频的世界模型使像素级环境在相机层面具备了交互性：用户可自由调整视角，模型则同步生成连贯的视觉延续。然而，其动作空间仍不完整——用户虽能控制相机，却无法对单个物体施以操作。由于真实世界的交互本质上是物体中心的，这类模型更像是被动的场景观察者，而非真正可操作的虚拟环境。我们提出WorldCraft框架，将交互式视频世界模型从相机导航扩展至物体级轨迹动作。给定用户点击和手绘路径后，WorldCraft可生成未来帧，使选定物体沿预定轨迹运动，同时相机继续自由探索场景。该框架通过轨迹中心控制管线实现：首先，归一化世界轨迹（NWT）以相机无关的世界坐标系表征用户绘制的运动，并动态将其重投影至当前相机位姿，从而分离物体运动与相机引发的屏幕空间位移；随后，空间路径LoRA（SP-LoRA）将此世界空间信号注入模型的空间控制通路，在保留预训练相机控制器能力的同时新增物体操控功能；最后，轨迹锚定状态持久化（TASP）将世界轨迹视为持久空间状态，在轨迹条件生成后刷新自回归记忆，使移出相机视野的物体在重新出现时仍处于更新后的位置。实验表明，WorldCraft能实现精确的物体控制，在仅使用相机评估时保持基于视频的世界模型的相机保真度，并能在长程自回归推理及物体短暂离屏的情况下维持物体状态一致性。

English

Recent video-based world models have made pixel-space environments interactive at the camera level: users can navigate viewpoints while the model generates coherent visual continuations. Yet their action spaces remain incomplete: users can move the camera, but cannot act on individual objects. Since real-world interaction is inherently object-centric, such models remain closer to passive scene observers than truly manipulable environments. We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions. Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory while the camera continues to navigate the scene. WorldCraft achieves this through a trajectory-centric control pipeline: First, Normalized World Trajectory (NWT) represents user-drawn motion in a camera-invariant world coordinate system and dynamically re-projects it under the current camera pose, separating object motion from camera-induced screen-space displacement; Spatial-Pathway LoRA (SP-LoRA) then injects this world-space signal through the model's spatial-control pathway, adding object manipulation capability while preserving the pretrained camera controller; finally, Trajectory-Anchored State Persistence (TASP) treats the world trajectory as a persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation, allowing moved objects to reappear at their updated positions after leaving the camera view. Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts with off-camera excursions.