將圖像作為集合進行分詞

Tokenize Image as a Set

March 20, 2025

作者: Zigang Geng, Mengde Xu, Han Hu, Shuyang Gu

cs.AI

摘要

本文提出了一種基於集合的標記化與分佈建模的全新圖像生成範式。與傳統方法將圖像序列化為固定位置的潛在代碼並採用統一壓縮率不同，我們引入了一種無序的標記集合表示法，根據區域語義複雜度動態分配編碼容量。這種TokenSet增強了全局上下文聚合，並提高了對局部擾動的魯棒性。為解決建模離散集合的關鍵挑戰，我們設計了一種雙重轉換機制，將集合雙射轉換為具有求和約束的固定長度整數序列。此外，我們提出了固定和離散擴散框架——這是首個同時處理離散值、固定序列長度和求和不變性的框架，實現了有效的集合分佈建模。實驗結果表明，我們的方法在語義感知表示和生成質量方面具有顯著優勢。我們在新型表示與建模策略上的創新，推動了視覺生成超越傳統的序列標記範式。我們的代碼和模型已公開於https://github.com/Gengzigang/TokenSet。

English

This paper proposes a fundamentally new paradigm for image generation through set-based tokenization and distribution modeling. Unlike conventional methods that serialize images into fixed-position latent codes with a uniform compression ratio, we introduce an unordered token set representation to dynamically allocate coding capacity based on regional semantic complexity. This TokenSet enhances global context aggregation and improves robustness against local perturbations. To address the critical challenge of modeling discrete sets, we devise a dual transformation mechanism that bijectively converts sets into fixed-length integer sequences with summation constraints. Further, we propose Fixed-Sum Discrete Diffusion--the first framework to simultaneously handle discrete values, fixed sequence length, and summation invariance--enabling effective set distribution modeling. Experiments demonstrate our method's superiority in semantic-aware representation and generation quality. Our innovations, spanning novel representation and modeling strategies, advance visual generation beyond traditional sequential token paradigms. Our code and models are publicly available at https://github.com/Gengzigang/TokenSet.