Molt：一个可扩展的、原生PyTorch的智能体强化学习训练框架

摘要

主体性强化学习研究涉及持续的算法修改、新估计量、新流水线阶段、新展开方案，而在主流框架中，每一次改动都需要贯穿训练器、分布式后端和展开粘合层：每一次迭代的成本都落在研究者身上。Molt是一个基于PyTorch原生的训练框架，旨在保持低成本：代码库紧凑且清晰，足以让研究者牢记于心，也便于AI编程助手完整阅读并推理，从而能够端到端追踪和调整算法流程。主体是一个普通程序，通过一个异步循环训练多模态和混合专家策略，且永远不会训练一个未由其生成的token，在token、策略版本和模型语义上保持一致。简洁并非以性能为代价：在匹配的完全异步协议下，Molt在统计性能上可与基于Megatron的最先进架构相媲美。Molt为开源项目，提供配方和容器，访问地址为https://github.com/NVIDIA-NeMo/labs-molt。

English

Agentic reinforcement learning research is constant algorithm modification, new estimators, new pipeline stages, new rollout schemes, and in mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue: the cost lands on the researcher at every iteration. Molt is a PyTorch-native training framework built to keep that cost small: a codebase compact and clean enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety, so the algorithm flow can be traced and changed end to end. The agent is an ordinary program, and one asynchronous loop trains multimodal and mixture-of-experts policies while never training on a token it did not generate, consistent in tokens, policy versions, and model semantics. Leanness does not cost performance: under a matched, fully asynchronous protocol, Molt is statistically comparable to a state-of-the-art Megatron-based stack. Molt is open source and provides recipes and containers at https://github.com/NVIDIA-NeMo/labs-molt.