DumpsterCluster：從垃圾堆尋寶到在60美元GPU上服務LLaMA-70B

摘要

隨著AI資料中心汰換仍具功能的GPU，大量效能尚可的加速器進入二手市場。本文探討這些退役GPU是否能找到具生產力的第二生命，形成可服務現代LLM推論的「垃圾場叢集」（DumpsterCluster），以及在何種條件下此類再利用在經濟上可行且環境上可永續。我們從零開始僅使用二手零件實際建構了一個128-GPU的DumpsterCluster，並運行了整整一年。以當前市場價格而言（DumpsterCluster約2.2萬美元，相較於8-GPU B200系統的60萬美元），其經濟優勢相當可觀。透過管線平行（pipeline-parallel）最佳化，我們以V100為基礎的DumpsterCluster在LLaMA-70B推論吞吐量上達到具競爭力的表現，驗證了實際生產應用的可行性。然而，我們的部署揭示了關鍵的脈絡依賴性。較舊世代的GPU每個token消耗顯著更多的能源，使得總擁有成本僅在電價低廉的地區才具優勢。在電網平均碳強度下，二手系統在8B模型上每token產生的總碳排放量約為當前世代硬體的4倍，在70B模型上更高達40倍以上。這些發現顯示GPU的第二生命並非普遍可永續——硬體再利用必須與低碳能源策略性地結合。當部署在能源經濟效益良好且電力潔淨的地區時，二手GPU提供了擴展AI算力的可行途徑，同時促進可負擔性、能源安全與環境責任。

English

As AI datacenters retire functional GPUs, vast quantities of still capable accelerators enter secondary markets. This paper investigates whether these retired GPUs can find a productive afterlife to form a DumpsterCluster that can serve modern LLM inference, and under what conditions such repurposing is economically viable and environmentally sustainable. We physically built a 128-GPU DumpsterCluster from scratch using only second-hand components and ran it for one year. At current market prices (\22K for the DumpsterCluster vs. 600K for an 8-GPU B200 system), the economic advantages are substantial. Through pipeline-parallel optimizations, our V100 based DumpsterCluster achieves competitive LLaMA-70B throughput, validating production viability. However, our deployment reveals critical context dependencies. Older GPUs consume significantly more energy per token, making total cost of ownership favorable only in regions with inexpensive electricity. Under grid-average carbon intensity, second-hand systems can produce approximately 4x higher total carbon emissions per token for 8B models, and over 40x for 70B models, compared to current-generation hardware. These findings show that GPU afterlife is not universally sustainable - hardware repurposing must be strategically coupled with low carbon energy sources. When deployed in regions with favourable energy economics and clean electricity, second-hand GPUs offer a viable pathway for expanding AI capacity while advancing affordability, energy security, and environmental responsibility.