三元組塊擴散RWKV

摘要

因果變換器語言模型受制於嚴格的序列解碼與二次方的每步注意力成本。雖然線性時間因果模型與離散擴散模型各自解決了這些弱點，但它們的整合本質上存在矛盾：擴散需要雙向注意力，而因果模型僅為單向。為統合這兩種架構，我們提出B³D-RWKV，一種擴散型RWKV變體，透過三元組區塊佈局方法，將模型的O(L)推論效率與並行、雙向的離散擴散機制相結合。B³D-RWKV-7.2B在八項任務測試套件中達到與現有模型相當的準確度，同時在解碼吞吐量上顯著超越基準模型，平均加速1.6倍。

English

Causal Transformer language models suffer from strictly sequential decoding and a quadratic per-step attention cost. While linear-time causal models and discrete diffusion models each address these weaknesses, their integration remains inherently inconsistent: diffusion requires bidirectional attention, while causal models are unidirectional. To unify these architectures, we propose B^3D-RWKV, a diffusion RWKV variant that integrates the model's O(L) inference efficiency with parallel, bidirectional discrete-diffusion through a triplet-block layout method. B^3D-RWKV-7.2B reaches comparable accuracy on an 8-task suite versus existing models while significantly outperforming baselines in decoding throughput with an average of 1.6times speedup.