三元组块扩散RWKV

摘要

因果Transformer语言模型受限于严格顺序解码和每次注意力步骤的二次方成本。尽管线性时间因果模型和离散扩散模型各自解决了这些弱点，但它们的整合存在固有矛盾：扩散需要双向注意力，而因果模型是单向的。为统一这些架构，我们提出B³D-RWKV，这是一种扩散RWKV变体，通过三元组块布局方法，将模型的O(L)推理效率与并行双向离散扩散相结合。B³D-RWKV-7.2B在8任务套件上达到了与现有模型相当的准确率，同时在解码吞吐量上显著优于基线模型，平均加速1.6倍。

English

Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose B^3D-RWKV, a diffusion RWKV variant that integrates the model's O(L) inference efficiency with parallel, bidirectional discrete-diffusion through a triplet-block layout method. B^3D-RWKV-7.2B reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of 1.6times speedup.