单步梯度延迟并非大规模异步流水线并行大语言模型预训练的障碍

摘要

现代大规模LLM预训练得益于流水线并行的应用，但同步实现会导致流水线气泡期间GPU空闲，造成计算资源浪费。异步流水线并行消除了这些气泡，以梯度陈旧性为代价最大化吞吐量。在异步调度方案中，PipeDream-2BW尤为引人注目：不同于原始PipeDream调度，它能够确保梯度延迟恒定为单步，与流水线深度无关。然而，由于普遍认为在陈旧性条件下优化本质上不稳定，该方法的采用仍十分有限。本文挑战这一假设，证明单步延迟下的性能退化主要取决于优化器选择，而非固有局限性。我们首次提供了全面的实证分析，表明尽管PipeDream-2BW提出时的主流优化器AdamW确实遭受严重退化，但近期方法（如Muon）在单步延迟下展现出强鲁棒性。我们引入了一种与优化器无关的、基于误差反馈的修正方案，以进一步缓解延迟影响。我们提供了支持性理论分析，证明了Muon在有无该修正下的收敛性。针对多达100亿参数模型的广泛评估证实，我们的策略弥合了与同步训练的性能差距，突显了异步流水线并行在大规模场景下的实际潜力。

English

Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parallelism eliminates these bubbles, maximizing throughput at the cost of gradient staleness. Among asynchronous schedules, PipeDream-2BW is particularly appealing: unlike the original PipeDream schedule, it ensures a constant one-step gradient delay regardless of pipeline depth. However, its adoption remains limited due to the common belief that optimizing under staleness is fundamentally unstable. In this work, we challenge this assumption, demonstrating that degradation under one-step delay depends strongly on optimizer choice rather than being an intrinsic limitation. We provide the first comprehensive empirical analysis showing that while AdamW, the predominant optimizer at the time when PipeDream-2BW was introduced, indeed suffers from severe degradation, recent methods like Muon exhibit strong robustness under a one-step delay. We introduce an optimizer-agnostic Error Feedback-inspired correction to further mitigate delay effects. We provide supporting theoretical analysis demonstrating convergence for Muon with and without this correction. Extensive evaluation on models up to 10B parameters confirms that our strategies bridge the performance gap with synchronous training, highlighting the practical potential of asynchronous pipeline parallelism at scale.