Molt：一個可擴展的原生PyTorch訓練框架，用於智能體強化學習

摘要

自主性強化學習研究涉及持續的演算法修改、新的估計器、新的管線階段、新的推展方案，而在主流框架中，每次變更都貫穿於訓練器、分散式後端及推展銜接層之中：這些成本每次迭代都落在研究者身上。Molt 是一個以 PyTorch 原生設計的訓練框架，旨在將此成本降至最低：其程式碼庫精簡且清晰，足以讓研究者牢記於心，也讓 AI 輔助程式碼工具能完整閱讀與推理，從而實現演算法流程的端到端追蹤與修改。代理程式是一個普通程序，透過一個非同步迴圈訓練多模態及混合專家策略，且從不對其未生成的 token 進行訓練，確保 token、策略版本與模型語意的一致性。輕量不損效能：在匹配的完全非同步協議下，Molt 的統計表現與基於 Megatron 的頂尖架構相當。Molt 為開源，並在 https://github.com/NVIDIA-NeMo/labs-molt 提供配方與容器。

English

Agentic reinforcement learning research is constant algorithm modification, new estimators, new pipeline stages, new rollout schemes, and in mainstream frameworks each change threads through layers of trainer, distributed backend, and rollout glue: the cost lands on the researcher at every iteration. Molt is a PyTorch-native training framework built to keep that cost small: a codebase compact and clean enough for a researcher to hold in their head, and for an AI coding assistant to read and reason about in its entirety, so the algorithm flow can be traced and changed end to end. The agent is an ordinary program, and one asynchronous loop trains multimodal and mixture-of-experts policies while never training on a token it did not generate, consistent in tokens, policy versions, and model semantics. Leanness does not cost performance: under a matched, fully asynchronous protocol, Molt is statistically comparable to a state-of-the-art Megatron-based stack. Molt is open source and provides recipes and containers at https://github.com/NVIDIA-NeMo/labs-molt.