改進的大型語言擴散模型

摘要

現代大型語言模型主要採用自迴歸分解與因果注意力進行訓練。我們提出 iLLaDA，一個從零開始訓練、具備完全雙向注意力的 8B 遮罩擴散語言模型。iLLaDA 在預訓練與監督式微調（SFT）過程中持續使用遮罩擴散目標，將預訓練規模擴展至 12T 詞元，並在 25B 詞元的指令語料庫上進行 12 個週期的微調。我們進一步採用可變長度生成以提升效率，並引入基於置信度的評分機制用於多選題評估。相較於 LLaDA，iLLaDA 在一般基準、數學與程式碼評測中均有顯著提升；例如，iLLaDA-Base 在 BBH 上提升 21.6 分，在 ARC-Challenge 上提升 14.9 分；而 iLLaDA-Instruct 在 MATH 上提升 14.5 分，在 HumanEval 上提升 16.5 分。儘管採用非自迴歸訓練，iLLaDA 在多項基準中仍能與 Qwen2.5 7B 競爭。這些結果顯示，從零開始的全雙向擴散訓練是一條邁向強大語言模型的競爭力路徑。模型權重與程式碼：https://github.com/ML-GSAI/LLaDA。

English

Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present iLLaDA, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.