Puppet-Master：將互動式影片生成擴展為運動先驗，用於部件級別動態。

Puppet-Master: Scaling Interactive Video Generation as a Motion Prior for Part-Level Dynamics

August 8, 2024

作者: Ruining Li, Chuanxia Zheng, Christian Rupprecht, Andrea Vedaldi

cs.AI

摘要

我們提出了 Puppet-Master，一個互動式影片生成模型，可作為部分動態的運動先驗。在測試時，給定單張圖像和稀疏的運動軌跡集（即拖曳），Puppet-Master 能夠合成一段影片，展現忠實於給定拖曳交互作用的逼真部分層級運動。這是通過對一個大規模預訓練的影片擴散模型進行微調來實現的，我們提出了一種新的條件架構，以有效注入拖曳控制。更重要的是，我們引入了全對第關注機制，這是廣泛採用的空間關注模塊的可替換方案，通過解決現有模型中的外觀和背景問題，顯著提高了生成質量。與其他在野外影片上訓練並主要移動整個對象的運動條件影片生成器不同，Puppet-Master 是從 Objaverse-Animation-HQ 學習的，這是一個經過精心策劃的部分層級運動片段新數據集。我們提出了一種策略，可以自動過濾出次優動畫並用有意義的運動軌跡增強合成渲染。Puppet-Master 在各種類別的真實圖像上有很好的泛化能力，在真實世界基準測試中以零樣本方式優於現有方法。請參閱我們的項目頁面以獲取更多結果：vgg-puppetmaster.github.io。

English

We present Puppet-Master, an interactive video generative model that can serve as a motion prior for part-level dynamics. At test time, given a single image and a sparse set of motion trajectories (i.e., drags), Puppet-Master can synthesize a video depicting realistic part-level motion faithful to the given drag interactions. This is achieved by fine-tuning a large-scale pre-trained video diffusion model, for which we propose a new conditioning architecture to inject the dragging control effectively. More importantly, we introduce the all-to-first attention mechanism, a drop-in replacement for the widely adopted spatial attention modules, which significantly improves generation quality by addressing the appearance and background issues in existing models. Unlike other motion-conditioned video generators that are trained on in-the-wild videos and mostly move an entire object, Puppet-Master is learned from Objaverse-Animation-HQ, a new dataset of curated part-level motion clips. We propose a strategy to automatically filter out sub-optimal animations and augment the synthetic renderings with meaningful motion trajectories. Puppet-Master generalizes well to real images across various categories and outperforms existing methods in a zero-shot manner on a real-world benchmark. See our project page for more results: vgg-puppetmaster.github.io.