DreamTraj：通过读取未渲染视频扩散潜变量生成6自由度物体轨迹

摘要

在操作过程中，准确预测物体轨迹对于形成感知-行动闭环至关重要。目前进展受限于两个方面：现有数据集缺乏细粒度的语言到运动标注，而现有预测器要么依赖视频、深度或CAD模型等特权输入，要么通过代价高昂且易出错的感知流水线从完全生成的视频中恢复运动。我们通过MOVE数据集填补了这一监督空白，该数据集包含5,038条以物体为中心的第一人称轨迹，每条轨迹都配有细粒度的自然语言指令，而非粗粒度的动词-名词标签。我们进一步提出DreamTraj，它仅凭单张RGB图像和一条任务指令即可预测物体6-DoF轨迹，推理时无需视频、深度或CAD模型：DreamTraj并非生成视频，而是在早期去噪步骤中从冻结的图像到视频扩散模型的内部表示中读取运动信息。一个轻量级的流匹配读取器将查询-键注意力轨迹与池化隐藏状态解码为相对6-DoF位姿。据我们所知，这是首个直接从视频扩散模型的中间表示而非生成像素中解码物体6-DoF轨迹的方法。在平移和旋转方面，DreamTraj均超越了使用多帧或特权输入的预测器，达到了新的最先进水平，且运行速度是先生成后提取流水线的4.6倍。

English

Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.