序列缩放假说

摘要

尽管机器学习通过大规模并行化取得了进展，但我们发现了一个关键盲点：某些问题本质上是顺序性的。这些“固有串行”问题——从数学推理到物理模拟再到序列决策——需要依赖性的计算步骤，无法并行化。借鉴复杂性理论，我们形式化了这一区别，并证明当前以并行为中心的架构在此类任务上面临根本性限制。我们认为，认识到计算的串行性质对机器学习、模型设计和硬件开发具有深远影响。随着人工智能应对日益复杂的推理，有意识地扩展串行计算——而不仅仅是并行计算——对于持续进步至关重要。

English

While machine learning has advanced through massive parallelization, we identify a critical blind spot: some problems are fundamentally sequential. These "inherently serial" problems-from mathematical reasoning to physical simulations to sequential decision-making-require dependent computational steps that cannot be parallelized. Drawing from complexity theory, we formalize this distinction and demonstrate that current parallel-centric architectures face fundamental limitations on such tasks. We argue that recognizing the serial nature of computation holds profound implications on machine learning, model design, hardware development. As AI tackles increasingly complex reasoning, deliberately scaling serial computation-not just parallel computation-is essential for continued progress.

序列缩放假说

The Serial Scaling Hypothesis

摘要

Support