序列縮放假說

摘要

儘管機器學習已通過大規模並行化取得進展，我們發現了一個關鍵盲點：某些問題本質上是順序性的。這些“固有序列”問題——從數學推理到物理模擬再到序列決策——需要依賴性的計算步驟，這些步驟無法並行化。借鑒複雜性理論，我們形式化了這一區分，並證明當前的並行中心架構在此類任務上面臨根本性限制。我們認為，認識到計算的序列性質對機器學習、模型設計和硬件開發具有深遠影響。隨著人工智能處理日益複雜的推理，有意擴展序列計算——而不僅僅是並行計算——對於持續進步至關重要。

English

While machine learning has advanced through massive parallelization, we identify a critical blind spot: some problems are fundamentally sequential. These "inherently serial" problems-from mathematical reasoning to physical simulations to sequential decision-making-require dependent computational steps that cannot be parallelized. Drawing from complexity theory, we formalize this distinction and demonstrate that current parallel-centric architectures face fundamental limitations on such tasks. We argue that recognizing the serial nature of computation holds profound implications on machine learning, model design, hardware development. As AI tackles increasingly complex reasoning, deliberately scaling serial computation-not just parallel computation-is essential for continued progress.

序列縮放假說

The Serial Scaling Hypothesis

摘要

Support