Miles v0.1：生產級後訓練

摘要

我們提出 Miles v0.1，一套面向前沿後訓練的全端、可投入生產系統。奠基於 slime 的簡潔設計，Miles 在強化學習（RL）訓練迴路的每個階段皆圍繞單一原則進行設計：元件應可驗證、簡潔且可自訂。Miles 以準確性、效率、可靠性與可擴展性為首要目標，旨在讓研究人員與企業皆能取用前沿規模的 RL。本報告端到端說明該系統：以 SGLang 為基礎建構的 rollout 引擎、可選擇兩種後端（NVIDIA Megatron-LM 與 PyTorch FSDP）的訓練器，以及針對不同部署拓撲的三種權重同步傳輸方式。除全參數 RL 外，Miles 亦支援 LoRA 強化學習、同策略蒸餾、監督式微調，以及真·同策略 rollout 訓練對齊，並將相同架構延伸至擴散模型。文末以一個端到端案例研究作結：在終端機操作程式設計任務上，對 GLM-5.2 744B-A40B 模型進行全非同步的代理式 RL，於 64 張 NVIDIA GB300 GPU 上運行，前 30 個量測步驟的步進時間中位數為 263 秒。Miles 已在 https://github.com/radixark/miles 開源，專案網站為 https://miles.radixark.com。

English

We present Miles v0.1, a full-stack, production-ready system for frontier post-training. Building upon the clean design of slime, Miles designs each stage of the reinforcement-learning (RL) training loop around a single principle: components should be verified, clean, and customizable. With accuracy, efficiency, reliability, and scalability as first-class goals, Miles aims to make frontier-scale RL accessible to researchers and enterprises alike. This report walks through the system end to end: rollout engines built on SGLang, a trainer with a choice of two backends (NVIDIA Megatron-LM and PyTorch FSDP), and three weight-synchronization transports for different deployment topologies. Beyond full-parameter RL, Miles also supports LoRA RL, on-policy distillation, supervised fine-tuning, and true-on-policy rollout-training alignment, and extends the same architecture to diffusion models. We close with an end-to-end case study: fully asynchronous agentic RL on a GLM-5.2 744B-A40B model over terminal-use coding tasks, running on 64 NVIDIA GB300 GPUs with a median step time of 263 seconds over the first 30 measured steps. Miles is open-sourced at https://github.com/radixark/miles, with the project website at https://miles.radixark.com.