用于自由形式交错文本-图像生成的阐明统一多模态模型

摘要

生成文本与图像的生成式AI模型的进步，标志着多模态智能领域迈出关键一步，尤其适用于涉及两种模态交错的任务。为将这一智能推向下一阶段，模型需能自主生成自由形式的交错图文序列。本文提出ILLUME-X，一种先进的统一多模态范式，通过提升多模态数据效率并稳定多模态训练过程，实现高质量的自由形式图文交错生成。ILLUME-X包含三大核心组件：（i）专为交错图文生成优化的扩展训练数据管道；（ii）针对自由长度多模态标记序列的自适应目标渐进式训练策略；（iii）用于评估交错图文序列的客观全面评估方法ILScore。值得注意的是，我们的ILLUME-X在风格转换、图像分解和故事叙述等多项交错图文生成任务中，均优于先前统一模型。

English

The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.