遮蔽擴散語言模型是適用於自主強化學習的強大且可操控的基於文本的世界模型。

摘要

近年來，強化學習（RL）的發展凸顯了對多樣化、專業化訓練環境的需求。隨著模型性能的提升，由人工精心設計、具有固定任務與獎勵難度的環境逐漸失去有效信號，而長序列中的稀疏獎勵則會導致模型在特定工作流程或工具結構上產生模式崩潰。能模擬環境狀態的世界模型已達到與純隨機展開相當的性能，使其在按需擴展多樣性方面極具潛力。然而，自迴歸（AR）世界模型存在從左到右的偏誤，無法對全局相互依賴的狀態錨點（如工具結構、先前回合與預期結果）進行條件化處理。我們（i）將基於文本的世界建模形式化為一個可引導的轉移動態問題，分解為初始狀態、任務上下文、工具結構、領域規則與引導指令；（ii）整理出跨越九個開源環境與十二個前沿模型家族的239,403條基於狀態-動作軌跡的數據。我們比較了自迴歸語言模型（AR LM）與遮罩擴散語言模型（MDLM），結果顯示MDLM透過雙向錨點感知去噪，在連貫性、基礎紮實度與經驗驗證的展開多樣性上，均優於參數量達四倍之多的LLM，且推理延遲相近。我們引入了一套具確定性狀態檢查的即插即用GRPO訓練框架，並在三個分佈外環境（ScienceWorld、ALFWorld、AppWorld）中，針對三種1.2B至7B參數規模的智能體主幹（LFM2.5、Qwen3、Mistral）進行零樣本遷移消融實驗，在不經環境特定微調的情況下，相較基線模型取得了高達47%的絕對性能提升。我們進一步對對抗情境下的失敗模式進行行為分析，並針對真實感、結果正確性與訓練實用性進行人類評估。我們開源此項工作，以鼓勵此方向的研究。

English

Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressive (AR) world models suffer from a left-to-right bias preventing conditioning on globally interdependent state anchors such as tool schemas, prior turns, and expected outcomes. We (i) formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state, task context, tool schemas, domain rules, and steering directives, and (ii) curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families. We compare AR LMs and masked diffusion language models (MDLMs), showing MDLMs, via bidirectional anchor-aware denoising, achieve better coherence, groundedness, and empirically validated rollout diversity than LLMs over 4x their parameter size, at comparable inference latency. We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments (ScienceWorld, ALFWorld, AppWorld) across three 1.2B-7B agent backbones (LFM2.5, Qwen3, Mistral), achieving up to 47% absolute gains over baselines without environment-specific fine-tuning. We further conduct behavioral analysis of failure modes under adversarial scenarios and human evaluation on realism, outcome correctness, and training utility. We open-source our work to encourage research in this direction.