DreamTraj：透過讀取未渲染的影片擴散潛在特徵生成6自由度物體軌跡

摘要

在操作過程中準確預測物體軌跡，對於閉合感知-行動迴路至關重要。然而，進展在兩個方面受到限制：現有數據集缺乏細粒度的語言到動作標註，而現有的預測器要麼依賴特權輸入（如影片、深度或CAD模型），要麼透過昂貴且容易出錯的感知管線從完全生成的影片中恢復動作。我們透過MOVE數據集填補了監督差距，該數據集包含5,038條以物體為中心的自我中心軌跡，每一條都配有細粒度的自然語言指令，而非粗略的動詞-名詞標籤。我們進一步提出DreamTraj，它從單張RGB影像和任務指令預測六自由度物體軌跡，在推論時不需要影片、深度或CAD模型：它並非生成影片，而是在早期去噪步驟中，從凍結的影像到影片擴散模型的內部表示中讀取動作。一個輕量級流匹配讀取器將查詢-鍵注意力軌跡和池化隱藏狀態解碼為相對六自由度姿態。據我們所知，這是第一種直接從影片擴散模型的中間表示解碼物體六自由度軌跡、而非從生成像素中解碼的方法。DreamTraj在對抗使用多幀或特權輸入的預測器時，在平移和旋轉方面均創下了新的最佳表現，並且運行速度比先生成再提取的管線快4.6倍。

English

Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.