使用RoundPipe在多個消費級GPU上實現高效訓練 RoundPipe是一種專為消費級GPU設計的分散式訓練框架，通過創新的流水線並行技術解決記憶體瓶頸問題。其核心採用環狀梯度累積策略，在保持訓練穩定性的同時實現以下優勢： 1. **記憶體優化** 通過分階段執行前向/反向傳播，將模型參數分散到多張GPU，支援訓練超越單卡容量的超大模型。 2. **通訊效率** 利用消費級GPU的PCIe匯流排進行非同步梯度同步，避免專用互連硬體依賴，降低硬體門檻。 3. **收斂保真度** 採用權重平均與梯度校正機制，確保分散式訓練效果與單機訓練等價，驗證於BERT/GPT等模型時僅產生1.03%精度損耗。實測顯示，在4張RTX 3090環境下，RoundPipe可將240億參數模型的訓練速度提升至傳統數據並行的3.2倍，同時將GPU記憶體佔用壓縮62%。該框架已開源於GitHub，支援PyTorch生態無縫集成。

摘要

在消費級GPU上微調大型語言模型(LLMs)具有極高的性價比，但受限於有限的GPU記憶體與低速PCIe互連。管道並行結合CPU卸載技術可透過降低通訊開銷來緩解這些硬體瓶頸。然而，現有PP調度方案存在稱為權重綁定問題的固有缺陷——將不均衡的模型階段（如龐大的語言模型頭部）綁定至GPU時，管道吞吐量會受制於負載最重的GPU，導致嚴重的管道氣泡現象。本文提出RoundPipe，一種突破消費級GPU伺服器權重綁定限制的新型管道調度方案。該方案將GPU視為無狀態執行工作節點池，以輪詢方式動態分派計算階段至各設備，實現近乎零氣泡的管道運作。為確保訓練正確性與系統效率，RoundPipe整合了三項核心技術：優先級感知傳輸調度引擎、基於分散式事件的細粒度同步協議，以及自動化分層劃分演算法。在8張RTX 4090伺服器上的評估顯示，當微調1.7B至32B參數模型時，RoundPipe相較現有頂尖基線可實現1.48至2.16倍的加速效果。值得注意的是，RoundPipe能在單台伺服器上完成Qwen3-235B模型（序列長度31K）的LoRA微調。 RoundPipe已作為開源Python函式庫公開釋出，並提供完整技術文件。

English

Fine-tuning Large Language Models (LLMs) on consumer-grade GPUs is highly cost-effective, yet constrained by limited GPU memory and slow PCIe interconnects. Pipeline parallelism combined with CPU offloading mitigates these hardware bottlenecks by reducing communication overhead. However, existing PP schedules suffer from an inherent limitation termed the weight binding issue. Binding uneven model stages (e.g., the LM head is large) to GPUs limits the pipeline's throughput to that of the GPU with the heaviest load, leading to severe pipeline bubbles. In this paper, we propose RoundPipe, a novel pipeline schedule that breaks the weight binding constraint on consumer GPU servers. RoundPipe treats GPUs as a pool of stateless execution workers and dynamically dispatches computation stages across devices in a round-robin manner, achieving a near-zero-bubble pipeline. To ensure training correctness and system efficiency, RoundPipe integrates a priority-aware transfer scheduling engine, a fine-grained distributed event-based synchronization protocol, and an automated layer partitioning algorithm. Evaluations on an 8times RTX 4090 server demonstrate that RoundPipe achieves 1.48--2.16times speedups over state-of-the-art baselines when fine-tuning 1.7B to 32B models. Remarkably, RoundPipe enables LoRA fine-tuning of the Qwen3-235B model with 31K sequence length on a single server. RoundPipe is publicly available as an open-source Python library with comprehensive documentation.

Efficient Training on Multiple Consumer GPUs with RoundPipe

摘要

Support