Goku: 百万级通用数据集与基于指令的视频编辑基准

摘要

现有的基于指令的视频编辑数据集通常聚焦于单一任务的外观编辑，难以满足真实场景中复杂的创作需求。为弥补这一差距，我们提出了Goku——一个包含200万高质量、指令对齐的视频编辑对的大规模数据集，这是首个将任务边界从基础外观编辑扩展到多任务与结构操控（例如对主体运动的精确控制）的数据集。针对这些复杂任务中固有的数据合成挑战，我们设计了一种高效的数据合成流程，将复杂编辑分解为可控的子问题，并在整个过程中引入渐进式过滤系统以确保数据可靠性。此外，我们在Goku上探索了最优网络结构，并提出了Goku-Edit。为深入理解复杂编辑指令，Goku-Edit采用多模态大语言模型（MLLM）作为文本编码器，并采用解耦的双分支设计：专用掩码分支处理结构控制，使主分支专注于外观渲染。我们还提出了一个全面的视频编辑基准Goku-Bench，包含1000个人工验证的测试用例和7项新颖的编辑专属评估指标。在Goku-Bench上的评估显示，Goku-Edit在指令遵循方面比其他开源模型提升高达8%。

English

Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.