UniSpace：統一視覺表徵與可擴展多模態建模

摘要

語義視覺編碼器已成為多模態理解與影像生成中語義條件化的核心視覺介面。然而，其最終的 token 會丟棄細粒度的視覺細節，導致像素重建效果不佳，並限制了它們在如影像生成與編輯等對重建敏感的任務中的應用。在這項工作中，我們探討是否能基於預訓練的語義 ViT，在單一的視覺表示空間中同時建模理解、生成與編輯。我們證明，語義 ViT 的凍結 Transformer 區塊並非本質上無法保留視覺細節。相反地，原始的 patch 參數化會將表示導向語義抽象，使得從最終 token 中難以恢復細粒度資訊。基於此觀察，我們提出 Patch Reparameterization（patch 重新參數化），它在保留原有語義路徑的同時，加入具重建感知的 patch 嵌入，為相同的凍結 ViT 區塊提供細粒度視覺資訊。所產生的統一表示在保留多模態理解的同時，也能實現高保真影像重建，並取得良好的重建—生成權衡。我們進一步將此表示擴展為 UniSpace，一個 8B 規模的 Mixture-of-Transformer-Experts 模型，能在同一個視覺空間中執行理解、生成與編輯，而無需額外的 VAE 路徑。系統層級的評測展示了實際的文字轉影像生成與基於指令的影像編輯，證明重新參數化的預訓練 ViT 可作為可擴展多模態建模的統一視覺介面。

English

Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introduce Patch Reparameterization, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding that provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation into UniSpace, an 8B Mixture-of-Transformer-Experts model that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as a unified visual interface for scalable multimodal modeling.