掩码扩散语言模型是用于智能体强化学习的强大且可操控的基于文本的世界模型。

摘要

近期强化学习（RL）的发展催生了对多样化、专业化训练环境的需求。随着模型性能提升，固定任务与奖励难度的手工定制环境逐渐失效；而长程稀疏奖励则会导致模型在特定工作流或工具结构上出现模式坍缩。能够模拟环境状态的世界模型已实现与纯环境回放相当的生成性能，有望按需扩展环境多样性。然而，自回归（AR）世界模型存在从左到右的生成偏差，难以对全局互依的状态锚点（如工具架构、先前交互轮次与预期结果）进行条件约束。为此，我们（i）将基于文本的世界建模形式化为一个可引导的转移动态问题，并将其分解为初始状态、任务上下文、工具架构、领域规则与引导指令五个组成部分；（ii）整理了涵盖九个开源环境和十二个前沿模型系列的239,403条基于状态-动作的轨迹数据。通过对比AR语言模型与掩码扩散语言模型（MDLM），我们发现MDLM凭借双向锚点感知去噪机制，在一致性、事实基础性及经验验证的生成多样性方面，均优于参数规模4倍于此的大型语言模型，且推理延迟相当。我们提出了一个即插即用的GRPO训练框架，并引入确定性状态检查机制；在三个OOD环境（ScienceWorld、ALFWorld、AppWorld）中，基于三个1.2B~7B规模的智能体基座模型（LFM2.5、Qwen3、Mistral）进行了零样本迁移消融实验，在不进行环境特定微调的情况下，相比基线方法实现了最高47%的绝对性能提升。我们进一步针对对抗场景下的失败模式进行了行为分析，并开展了关于真实性、结果正确性与训练效用的人类评估。为鼓励该方向的研究，我们开源了相关工作。

English

Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressive (AR) world models suffer from a left-to-right bias preventing conditioning on globally interdependent state anchors such as tool schemas, prior turns, and expected outcomes. We (i) formalize text-based world modeling as a steerable transition-dynamics problem decomposed into initial state, task context, tool schemas, domain rules, and steering directives, and (ii) curate 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families. We compare AR LMs and masked diffusion language models (MDLMs), showing MDLMs, via bidirectional anchor-aware denoising, achieve better coherence, groundedness, and empirically validated rollout diversity than LLMs over 4x their parameter size, at comparable inference latency. We introduce a plug-and-play GRPO training framework with deterministic state checks, and perform zero-shot transfer ablations on three OOD environments (ScienceWorld, ALFWorld, AppWorld) across three 1.2B-7B agent backbones (LFM2.5, Qwen3, Mistral), achieving up to 47% absolute gains over baselines without environment-specific fine-tuning. We further conduct behavioral analysis of failure modes under adversarial scenarios and human evaluation on realism, outcome correctness, and training utility. We open-source our work to encourage research in this direction.