Chimera：混合视觉扩散Transformer的设计与Chinchilla缩放

摘要

视觉生成日益需要高分辨率图像、长视频和多模态上下文，这使得全注意力的二次方成本变得难以承受。我们提出Chimera——一种具有原理性扩展策略的混合视觉扩散主干网络。Chimera在不使用位置编码的情况下，将文本、图像和视频令牌按单一光栅顺序流进行处理。它结合了具有O(N)复杂度的Kim Delta Attention（KDA）用于长上下文状态追踪、交错的多头潜在注意力（MLA）用于直接的全局交互，以及模态感知的短卷积用于局部时空上下文。稀疏混合专家（MoE）层在控制激活计算量的同时扩展了模型容量。为扩展这一异构架构，我们提出HeteroP——一种按模块划分的方案，根据每个张量的功能扇入和模型深度，在宽度和深度维度上迁移超参数。HeteroP产出一组一致调优的模型族，用于拟合Chinchilla风格的计算最优律，涵盖激活模型规模、训练令牌数量和图像-视频数据比例。在这些律的指导下，我们训练了一个具有20亿激活参数的110亿参数Chimera模型。实验表明三个结果。第一，以预训练扩散损失衡量，稠密主干网络的计算效率是对应的全注意力Wan-2.1 20亿基线的1.7倍，而完整系统达到7.3倍。第二，无需针对特定长度的微调，Chimera即可从5秒训练片段零样本外推至30秒视频，最后五秒的FID仅退化6.5%。第三，拟合的律表明，计算最优的图像预训练将计算量近乎均匀地分配给激活模型规模和训练令牌数量，而视频预训练在更高预算下则适度偏向模型规模。这些结果为设计和扩展高效长上下文扩散架构奠定了基础。

English

Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.