深入探討DeepSeek-V3：AI架構中的擴展挑戰與硬體反思

摘要

大型語言模型（LLMs）的快速擴展揭示了當前硬體架構的關鍵限制，包括記憶體容量、計算效率和互連頻寬的約束。DeepSeek-V3在2,048個NVIDIA H800 GPU上進行訓練，展示了硬體感知的模型協同設計如何有效應對這些挑戰，實現大規模的成本效益訓練和推理。本文深入分析了DeepSeek-V3/R1模型架構及其AI基礎設施，重點介紹了多頭潛在注意力（MLA）以提升記憶體效率、專家混合（MoE）架構以優化計算與通信的權衡、FP8混合精度訓練以充分發揮硬體潛力，以及多平面網路拓撲以最小化集群級網路開銷等關鍵創新。基於DeepSeek-V3開發過程中遇到的硬體瓶頸，我們與學術界和產業界的同行展開了更廣泛的討論，探討了未來硬體的潛在方向，包括精確的低精度計算單元、規模擴展與分散式收斂，以及低延遲通信結構的創新。這些見解強調了硬體與模型協同設計在滿足AI工作負載日益增長需求中的關鍵作用，為下一代AI系統的創新提供了實用的藍圖。

English

The rapid scaling of large language models (LLMs) has unveiled critical limitations in current hardware architectures, including constraints in memory capacity, computational efficiency, and interconnection bandwidth. DeepSeek-V3, trained on 2,048 NVIDIA H800 GPUs, demonstrates how hardware-aware model co-design can effectively address these challenges, enabling cost-efficient training and inference at scale. This paper presents an in-depth analysis of the DeepSeek-V3/R1 model architecture and its AI infrastructure, highlighting key innovations such as Multi-head Latent Attention (MLA) for enhanced memory efficiency, Mixture of Experts (MoE) architectures for optimized computation-communication trade-offs, FP8 mixed-precision training to unlock the full potential of hardware capabilities, and a Multi-Plane Network Topology to minimize cluster-level network overhead. Building on the hardware bottlenecks encountered during DeepSeek-V3's development, we engage in a broader discussion with academic and industry peers on potential future hardware directions, including precise low-precision computation units, scale-up and scale-out convergence, and innovations in low-latency communication fabrics. These insights underscore the critical role of hardware and model co-design in meeting the escalating demands of AI workloads, offering a practical blueprint for innovation in next-generation AI systems.

深入探討DeepSeek-V3：AI架構中的擴展挑戰與硬體反思

Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures

摘要

Summary

Support

Support