闡明用於自由形式交錯文字圖像生成的統一多模態模型

摘要

能夠生成文字與圖像的生成式AI模型之進展，標誌著多模態智能領域中的關鍵一步，特別是涉及兩種模態交錯的任務。為將此智能推進至下一階段，模型必須能自主生成自由形式的交錯式文字-圖像序列。在本文中，我們提出ILLUME-X，這是一種先進的統一多模態範式，透過提升多模態資料效率與穩定多模態訓練過程，實現高品質、自由形式的交錯式文字-圖像生成。ILLUME-X包含三大核心組件：(i) 為交錯式文字-圖像生成最佳化的擴展訓練資料管線，(ii) 針對自由長度多模態令牌序列的自適應目標漸進式訓練策略，以及(iii) 針對交錯式文字-圖像序列的客觀且全面評估方法ILScore。值得注意的是，我們的ILLUME-X在多項交錯式文字-圖像生成任務（如風格轉換、影像分解與故事敘述）中，均優於先前的統一模型。

English

The advancement of generative AI models capable of producing text and image marks a critical step forward in the realm of multimodal intelligence, particularly for tasks involving the interleaving of both modalities. To advance this intelligence to the next stage, it is crucial for models to autonomously generate free-form interleaved text-image sequences. In this paper, we introduce ILLUME-X, an advanced unified multimodal paradigm that enables high-quality, free-form interleaved text-image generation by improving multimodal data efficiency and stabilizing the multimodal training process. ILLUME-X comprises three key components: (i) an expanded training data pipeline optimized for interleaved text-image generation, (ii) a progressive training strategy with self-adaptive objectives for free-length multimodal token sequences, and (iii) an objective and comprehensive evaluation method ILScore for interleaved text-image sequences. Notably, our ILLUME-X outperforms previous unified models across multiple interleaved text-image generation tasks like style transfer, image decomposition and storytelling.