「隨軌而行：基於點追蹤的影片合成與動作控制」

摘要

电影制作要求精确的运动控制与参考图像合成——现有方法将这两项能力分开处理。基于点轨迹条件约束的图像到视频模型将内容插入限制在首帧，而参考图像到视频模型则缺乏对参考内容跨帧整合的细粒度时空控制。我们提出“随轨迹而动”（Go-with-the-Track），通过联合基于多张参考图像和参考锚定点轨迹的条件控制，将两种能力统一——该模型扩展了传统点轨迹，使其明确建立生成帧与参考图像之间的对应关系，从而实现对视频全程的精确合成与运动控制。为此，我们引入了空间感知点轨迹嵌入，通过坐标级MLP与时间池化操作，编码完整的点轨迹坐标序列。这种表示方法捕捉了每个点轨迹的空间特征（作为唯一标识符），同时嵌入相似度直接关联空间邻近性，增强了模型区分与关联点轨迹的能力。我们通过轻量级适配器将这些点轨迹注入视频扩散Transformer，解决了像素到补丁的分辨率失配问题，同时避免了朴素点轨迹下采样中固有的运动细节损失。我们采用混合训练策略，在动态、静态及合成场景视频数据集上联合训练，以提升运动可控性。实验表明，“随轨迹而动”在单一模型中实现了卓越的运动与参考控制，并解锁了新能力：基于多参考条件驱动的点轨迹合成视频生成，以及面向静态与动态场景的摄像机控制。项目页面：https://eyeline-labs.github.io/Go-with-the-Track/

English

Filmmaking demands precise motion control and reference image compositing -- capabilities that existing methods treat separately. Point-track-conditioned image-to-video models restrict content insertion to the first frame, while reference-to-video models lack fine-grained spatial-temporal control over how reference content integrates across frames. We present Go-with-the-Track, which unifies both capabilities by jointly conditioning on multiple reference images and reference-anchored point-tracks -- extending conventional point-tracks to explicitly establish correspondences between generated frames and reference images, thus enabling precise compositing and motion control throughout the video. To achieve this, we introduce spatially-aware point-track embeddings that encode the full sequence of point-track coordinates using a coordinate-wise MLP followed by temporal pooling. This representation captures the spatial characteristics of each point-track (serving as a unique identifier), while the embedding similarity correlates directly with spatial proximity, enhancing the model's ability to distinguish and associate point-tracks. We inject these point-track embeddings into a video diffusion transformer via a lightweight adapter, resolving the pixel-to-patch resolution mismatch while avoiding the substantial motion detail loss inherent in naive point-track subsampling. We use a hybrid training strategy to train jointly on dynamic, static, and synthetic scene video datasets to boost motion controllability. Experiments demonstrate that Go-with-the-Track achieves superior motion and reference control in a single model and enables new capabilities: multi-reference conditioned video generation with point-track driven compositing, as well as camera control for both static and dynamic scenes. Project Page: https://eyeline-labs.github.io/Go-with-the-Track/