文字模板令牌是擴散Transformer中的隱含語義寄存器

摘要

文本到图像扩散变换器（DiTs）联合处理文本和图像令牌，但其在去噪过程中的内部计算机制仍不明确。我们引入了一种针对现代大规模DiTs的因果可解释性框架，该框架将注意力分解与跨令牌跨度、注意力头及层级的定向干预相结合。通过该框架将提示内容令牌与结构模板令牌分离，我们发现结构令牌在编码器输出中携带的提示特定信息极少。然而令人意外的是，它们成为主导的图像到文本注意力汇聚点，并在DiT内部因果性地维持目标身份，充当隐式语义寄存器。研究表明，这些结构令牌间接获取这种身份：提示语义首先注入图像潜在表征，随后被读回模板令牌，而非直接从提示令牌传递。受上述发现启发，我们为DiTs设计了一种无需训练的剪枝规则。那些对提示令牌注意力最强的注意力头是可舍弃的，剪除它们可在GenEval指标上仅降低1.4分的情况下减少20%的注意力FLOPs。我们进一步揭示了DiT中生成计算如何跨注意力头和深度进行组织，将语义路由与视觉合成分离，并从身份形成、传播到精炼逐步推进。本工作不仅表明输入时编码语义的令牌未必是生成过程中维持语义的令牌，还为DiTs内部机制提供了因果视角。

English

Text-to-image diffusion transformers (DiTs) jointly process text and image tokens, yet their internal computation during denoising remains poorly understood. We introduce a causal interpretability framework for modern large-scale DiTs that combines attention decomposition with targeted interventions across token spans, heads, and layers. Using it to separate prompt-content tokens from structural template tokens, we find that the structural tokens carry little prompt-specific information at the encoder output. Yet surprisingly, they emerge as dominant image-to-text attention sinks and causally maintain object identity inside the DiT, acting as implicit semantic registers. We show that they acquire this identity indirectly, with prompt semantics first injected into the image latents and then read back into the template tokens rather than transferred directly from the prompt tokens. Inspired by the above findings, we design a training-free pruning rule for DiTs. Heads that attend most strongly to prompt tokens are dispensable, and pruning them removes 20% of attention FLOPs with only a 1.4-point drop on GenEval. We further reveal how generative computation in DiTs is organized across heads and depth, separating semantic routing from visual synthesis and progressing from identity formation to propagation and refinement. Our work not only reveals that the tokens encoding semantics at input need not be those that maintain it during generation, but also provides a causal view of internal mechanisms in DiTs.