現在該由誰主導解碼？追蹤可靠軌跡以集成遮蔽擴散語言模型

摘要

掩蔽擴散語言模型（MDLMs）已成為序列生成中的一種獨特典範。隨著MDLMs在能力與知識覆蓋範圍上日益多元，一個重要的問題是如何整合它們的知識。為此，我們首先探討MDLMs獨特的解碼動態。我們發現，成功的生成在與答案相關的位置上展現出穩定的信心動態，而不可靠的軌跡則常可透過注入來自其他模型的中間狀態來修正。根據此觀察，我們提出基於軌跡的迭代集成（TIE），這是一個知識融合框架，其中MDLMs反覆識別可靠的解碼軌跡並在模型間傳遞。TIE追蹤答案相關位置上的信心動態，以判斷哪個模型當下遵循更可靠的軌跡，並選擇性地將部分去噪序列跨模型轉移。由於處於較有前景軌跡上的模型常在去噪步驟之間變換，TIE允許不同模型在生成的不同階段貢獻互補的優勢。在多樣化推理任務上的強勁表現，以及我們的分析，均表明TIE為MDLM集成這一尚未充分探討的問題提供了實用方法。

English

Masked Diffusion Language Models (MDLMs) have emerged as a distinct paradigm for sequence generation. As MDLMs become diverse in capabilities and knowledge coverage, an important question is how to combine their knowledge. Toward this, we first investigate the unique decoding dynamics of MDLMs. We find that successful generations exhibit stable confidence dynamics over answer-relevant positions, while unreliable trajectories can often be corrected by injecting promising intermediate states from other models. Guided by this observation, we propose TIE (Trajectory-based Iterative Ensembling), a knowledge fusion framework in which MDLMs iteratively identify reliable decoding trajectories and relay them across models. TIE tracks confidence dynamics over answer-relevant positions to determine which model currently follows a more reliable trajectory and selectively transfers partially denoised sequences across models. As the model on the more promising trajectory often changes across denoising steps, TIE allows different models to contribute complementary strengths at different stages of generation. Strong performance across diverse reasoning tasks, along with our analyses, suggests that TIE offers a practical approach to the underexplored problem of MDLM ensembling.