InstructX：迈向基于多模态大语言模型指导的统一视觉编辑

InstructX: Towards Unified Visual Editing with MLLM Guidance

October 9, 2025

作者: Chong Mou, Qichao Sun, Yanze Wu, Pengze Zhang, Xinghui Li, Fulong Ye, Songtao Zhao, Qian He

cs.AI

摘要

随着多模态大语言模型（MLLMs）在视觉理解与推理方面取得显著进展，利用其提升扩散模型编辑性能的兴趣日益增长。尽管进展迅速，但多数研究缺乏对MLLM设计选择的深入分析。此外，在某些复杂任务如视频编辑中，MLLMs与扩散模型的整合仍面临挑战。本文提出InstructX，一个统一的图像与视频编辑框架。具体而言，我们全面研究了如何将MLLMs与扩散模型结合，以执行跨多样任务的指令驱动编辑。基于此研究，我们分析了统一建模中图像与视频的协作与差异。（1）我们展示，在图像数据上的训练能够无需显式监督即涌现出视频编辑能力，从而缓解了视频训练数据稀缺带来的限制。（2）通过融入模态特定的MLLM特征，我们的方法有效将图像与视频编辑任务统一于单一模型之内。大量实验证明，我们的方法能处理广泛的图像与视频编辑任务，并达到业界领先的性能水平。

English

With recent advances in Multimodal Large Language Models (MLLMs) showing strong visual understanding and reasoning, interest is growing in using them to improve the editing performance of diffusion models. Despite rapid progress, most studies lack an in-depth analysis of MLLM design choices. Moreover, the integration of MLLMs and diffusion models remains an open challenge in some difficult tasks, such as video editing. In this paper, we present InstructX, a unified framework for image and video editing. Specifically, we conduct a comprehensive study on integrating MLLMs and diffusion models for instruction-driven editing across diverse tasks. Building on this study, we analyze the cooperation and distinction between images and videos in unified modeling. (1) We show that training on image data can lead to emergent video editing capabilities without explicit supervision, thereby alleviating the constraints imposed by scarce video training data. (2) By incorporating modality-specific MLLM features, our approach effectively unifies image and video editing tasks within a single model. Extensive experiments demonstrate that our method can handle a broad range of image and video editing tasks and achieves state-of-the-art performance.