UniFork：探索模態對齊以實現統一的多模態理解與生成

UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation

June 20, 2025

作者: Teng Li, Quanfeng Lu, Lirui Zhao, Hao Li, Xizhou Zhu, Yu Qiao, Jun Zhang, Wenqi Shao

cs.AI

摘要

統一圖像理解與生成已成為多模態人工智慧中一個極具前景的範式。儘管近期取得進展，此類統一模型的最佳架構設計仍是一個開放性挑戰。在本研究中，我們首先分析了針對理解與生成任務的專用專家模型以及現有統一模型的模態對齊行為。我們的分析揭示了一個關鍵觀察：理解任務受益於網絡深度中逐步增強的模態對齊，這有助於構建語義信息以實現更好的理解；相比之下，生成任務呈現出不同的趨勢：模態對齊在淺層增加，但在深層減少以恢復空間細節。這些不同的對齊模式在完全共享的Transformer骨幹中產生了根本性衝突，其中統一的表示流通常導致兩個任務的性能折衷。基於這一發現，我們提出了UniFork，一種新穎的Y形架構，它在淺層共享跨任務表示學習，而在深層採用任務專用分支以避免任務干擾。這一設計有效地平衡了共享學習與任務專用化。通過大量消融實驗，我們證明UniFork始終優於傳統的完全共享Transformer架構，並實現了與專用模型相當或更優的性能。

English

Unified image understanding and generation has emerged as a promising paradigm in multimodal artificial intelligence. Despite recent progress, the optimal architectural design for such unified models remains an open challenge. In this work, we start by analyzing the modality alignment behaviors of task-specific expert models for understanding and generation, as well as current unified models. Our analysis reveals a crucial observation: understanding tasks benefit from a progressively increasing modality alignment across network depth, which helps build up semantic information for better comprehension; In contrast, generation tasks follow a different trend: modality alignment increases in the early layers but decreases in the deep layers to recover spatial details. These divergent alignment patterns create a fundamental conflict in fully shared Transformer backbones, where a uniform representational flow often leads to performance compromises across two tasks. Motivated by this finding, we introduce UniFork, a novel Y-shaped architecture that shares the shallow layers for cross-task representation learning, while employing task-specific branches in deeper layers to avoid task interference. This design effectively balances shared learning and task specialization. Through extensive ablation experiments, we demonstrate that Unifork consistently outperforms conventional fully shared Transformer architectures, and achieves performance on par with or better than task-specific models.