潛在流轉換器

Latent Flow Transformer

May 20, 2025

作者: Yen-Chen Wu, Feng-Ting Liao, Meng-Hsi Chen, Pei-Chen Ho, Farhang Nabiei, Da-shan Shiu

cs.AI

摘要

Transformer，作為大型語言模型（LLMs）的標準實現，通常由數十至數百個獨立層組成。雖然增加層數可以提升性能，但這種方法被質疑效率低下，尤其是考慮到擴散模型和基於流的模型在圖像生成領域所展現的連續層的優越性。我們提出了潛在流Transformer（LFT），它通過流匹配訓練的單一學習傳輸算子替換了一組層，在保持與原始架構兼容的同時實現了顯著的壓縮。此外，我們針對現有基於流的方法在保持耦合性方面的局限性，引入了流步進（FW）算法。在Pythia-410M模型上，採用流匹配訓練的LFT壓縮了24層中的6層，並優於直接跳過2層的情況（語言模型對數的KL散度為0.407對比0.529），證明了這一設計的可行性。當使用FW進行訓練時，LFT進一步將12層蒸餾為一層，同時將KL散度降低至0.736，超越了跳過3層的結果（0.932），顯著縮小了自迴歸生成與基於流生成範式之間的差距。

English

Transformers, the standard implementation for large language models (LLMs), typically consist of tens to hundreds of discrete layers. While more layers can lead to better performance, this approach has been challenged as far from efficient, especially given the superiority of continuous layers demonstrated by diffusion and flow-based models for image generation. We propose the Latent Flow Transformer (LFT), which replaces a block of layers with a single learned transport operator trained via flow matching, offering significant compression while maintaining compatibility with the original architecture. Additionally, we address the limitations of existing flow-based methods in preserving coupling by introducing the Flow Walking (FW) algorithm. On the Pythia-410M model, LFT trained with flow matching compresses 6 of 24 layers and outperforms directly skipping 2 layers (KL Divergence of LM logits at 0.407 vs. 0.529), demonstrating the feasibility of this design. When trained with FW, LFT further distills 12 layers into one while reducing the KL to 0.736 surpassing that from skipping 3 layers (0.932), significantly narrowing the gap between autoregressive and flow-based generation paradigms.