VFX創建者：具可控擴散變壓器的動畫視覺效果生成

VFX Creator: Animated Visual Effect Generation with Controllable Diffusion Transformer

February 9, 2025

作者: Xinyu Liu, Ailing Zeng, Wei Xue, Harry Yang, Wenhan Luo, Qifeng Liu, Yike Guo

cs.AI

摘要

製作魔法和幻覺是電影製作中最令人振奮的部分之一，視覺效果（VFX）是打造難忘電影體驗的強大動力。儘管最近生成人工智慧的進步推動了通用圖像和視頻合成的發展，可控制VFX生成領域仍相對未被充分探索。在這項工作中，我們提出了一種新的動畫VFX生成範式，即圖像動畫，其中動態效果是從易於使用的文本描述和靜態參考圖像生成的。 我們的工作主要有兩個貢獻：（i）Open-VFX，第一個高質量VFX視頻數據集，涵蓋了15個不同類別的效果，並附有文本描述、空間條件的實例分割遮罩和時間控制的起始-結束時間戳。 （ii）VFX Creator，一個簡單而有效的可控VFX生成框架，基於Video Diffusion Transformer。該模型包含一個空間和時間可控的LoRA適配器，需要極少的訓練視頻。具體而言，一個即插即用的遮罩控制模塊實現了實例級空間操作，而在擴散過程中嵌入的標記化起始-結束運動時間戳，以及文本編碼器，允許對效果的時間控制進行精確調節。 對Open-VFX測試集的廣泛實驗表明，所提出的系統在生成逼真和動態效果方面優越，實現了空間和時間可控性方面的最新性能和泛化能力。此外，我們引入了一個專門的指標來評估時間控制的精確性。通過將傳統VFX技術與生成方法相結合，VFX Creator為高效且高質量的視頻效果生成開啟了新的可能性，使先進的VFX技術更廣泛地可用於更廣泛的受眾。

English

Crafting magic and illusions is one of the most thrilling aspects of filmmaking, with visual effects (VFX) serving as the powerhouse behind unforgettable cinematic experiences. While recent advances in generative artificial intelligence have driven progress in generic image and video synthesis, the domain of controllable VFX generation remains relatively underexplored. In this work, we propose a novel paradigm for animated VFX generation as image animation, where dynamic effects are generated from user-friendly textual descriptions and static reference images. Our work makes two primary contributions: (i) Open-VFX, the first high-quality VFX video dataset spanning 15 diverse effect categories, annotated with textual descriptions, instance segmentation masks for spatial conditioning, and start-end timestamps for temporal control. (ii) VFX Creator, a simple yet effective controllable VFX generation framework based on a Video Diffusion Transformer. The model incorporates a spatial and temporal controllable LoRA adapter, requiring minimal training videos. Specifically, a plug-and-play mask control module enables instance-level spatial manipulation, while tokenized start-end motion timestamps embedded in the diffusion process, alongside the text encoder, allow precise temporal control over effect timing and pace. Extensive experiments on the Open-VFX test set demonstrate the superiority of the proposed system in generating realistic and dynamic effects, achieving state-of-the-art performance and generalization ability in both spatial and temporal controllability. Furthermore, we introduce a specialized metric to evaluate the precision of temporal control. By bridging traditional VFX techniques with generative approaches, VFX Creator unlocks new possibilities for efficient and high-quality video effect generation, making advanced VFX accessible to a broader audience.