扩散應進入語言模型的何處？幾何引導的隱藏狀態替換

摘要

連續擴散語言模型的表現仍落後於自回歸Transformer，部分原因在於擴散機制所應用的空間不適合語言去噪與Token恢復。我們提出DiHAL，一種幾何引導的擴散-Transformer混合模型，旨在探討擴散應如何進入預訓練Transformer。DiHAL透過幾何代理指標對層級進行評分，選取適合擴散的隱藏狀態介面，並以擴散步橋取代下層Transformer前綴，同時保留上層與原始語言模型輸出層。透過重建所選層的隱藏狀態而非Token，DiHAL避免了直接從連續域到離散域的恢復過程。在8B規模骨幹模型上的實驗顯示，在固定步橋訓練協議下，幾何評分能有效預測淺層插入位置；而在匹配擴散步/恢復訓練預算的診斷比較中，隱藏狀態恢復的表現優於連續擴散基準方法。這些結果表明，隱藏狀態的幾何特性有助於辨識預訓練語言模型中哪些位置適合進行基於擴散的取代。

English

Continuous diffusion language models lag behind autoregressive transformers, partly because diffusion is applied in spaces poorly suited to language denoising and token recovery. We propose DiHAL, a geometry-guided diffusion-transformer hybrid that asks where diffusion should enter a pretrained transformer. DiHAL scores layers with geometry-based proxies, selects a diffusion-friendly hidden-state interface, and replaces the lower transformer prefix with a diffusion bridge while retaining the upper layers and original LM head. By reconstructing the selected-layer hidden state rather than tokens, DiHAL avoids direct continuous-to-discrete recovery. Experiments on 8B-scale backbones show that the geometry score predicts effective shallow insertion layers under a fixed bridge-training protocol and that hidden-state recovery improves over continuous diffusion baselines in a diagnostic comparison matching the diffusion/recovery training budget. These results suggest that hidden-state geometry helps identify where diffusion-based replacement is feasible inside pretrained language models.