Chimera：混合視覺擴散Transformer之設計與Chinchilla規模化

摘要

視覺生成日益需要高解析度影像、長影片以及多模態上下文，使得全注意力機制的二次方成本變得難以承受。我們提出 Chimera，一種具有原則性擴展策略的混合視覺擴散主幹網路。Chimera 以單一光柵順序串流處理文字、影像與影片 Token，且無需位置編碼。它結合了具有 O(N) 複雜度的 Kimi Delta Attention（KDA）以進行長上下文狀態追蹤、交錯的 Multi-head Latent Attention（MLA）以實現直接的全域交互，以及模態感知的短卷積以捕捉局部時空脈絡。稀疏混合專家（MoE）層在控制啟用計算量的同時擴展了模型容量。為擴展此異構架構，我們提出 HeteroP，一種模組級的擴展方案，根據每個張量的功能性扇入與模型深度，在寬度與深度之間轉移超參數。HeteroP 產生了一組一致調校的模型家族，用以擬合 Chinchilla 風格的計算最優定律，涵蓋啟用模型規模、訓練 Token 數量以及影像-影片資料比例。在這些定律的引導下，我們訓練了一個具有 20 億啟用參數的 110 億參數 Chimera 模型。實驗結果顯示三項發現。第一，以預訓練擴散損失衡量，密集主幹網路的計算效率是配對的全注意力 Wan-2.1 2B 基線的 1.7 倍，而完整系統則達到 7.3 倍。第二，無需針對特定長度進行微調，Chimera 即可從 5 秒的訓練片段零樣本外推到 30 秒的影片，且最後五秒的 FID 僅退化 6.5%。第三，擬合的定律顯示，計算最優的影像預訓練將計算資源近乎均勻地分配給啟用模型規模與訓練 Token 數量，而影片預訓練在高預算下則略微偏好模型規模。這些結果為設計與擴展高效能的長上下文擴散架構奠定了基礎。

English

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.