Aurora：使用工具代理的統一影片編輯

摘要

近期影片編輯模型已收斂至統一的條件設計：單一擴散轉換器同時處理文字、來源影片及參考影像，並以同一組權重涵蓋替換、移除、風格轉換及參考驅動插入等任務。此設計具備靈活性，但前提是用戶已提供符合模型規格的文字、參考影像及局部編輯的空間定位，而實際需求往往缺乏這些資訊。我們提出 Aurora——一個智慧代理影片編輯框架，將工具增強型視覺語言模型代理與統一影片擴散轉換器結合。視覺語言模型代理會將原始用戶請求映射為符合擴散轉換器條件通道的結構化編輯計畫，從而在生成前解決文字與視覺層面的規格不足問題。我們透過監督式資料（涵蓋完整編輯規劃與參考影像選取）及偏好配對（用於強化工具使用與指令精煉）來訓練視覺語言模型代理。為評估智慧代理增強型影片編輯在文字與視覺規格不足情境下的表現，我們引入 AgentEdit-Bench 基準。在 AgentEdit-Bench 與兩個現有影片編輯基準上的實驗顯示，Aurora 較純指令基準方法有顯著提升，且視覺語言模型代理可遷移至相容的凍結式影片編輯模型。專案網頁：https://yeates.github.io/Aurora-Page

English

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page