OmniOpt：现代优化器的分类学、几何学与基准测试

摘要

大规模模型训练中的优化器选择已成为一个系统级的设计决策，受计算、内存、调优预算和任务多样性的共同约束，然而目前百余种方法的研究格局仍然碎片化。为此，我们提出OmniOpt——面向研究社区的优化器统一综述与基准实践指南。OmniOpt基于四个相互关联的组成部分。首先，我们将每次优化器更新视为通过五阶段元管道进行的结构化变换，并表明大多数方法仅涉及其中一两个阶段。其次，我们利用范数约束的线性最小化预言机（LMOs）来统一不同优化器。第三，这两种视角构成了一个双维度分类体系：一个维度将每种方法归入机制家族，另一个维度记录其旨在改进的可衡量训练目标。第四，也是本文的核心，我们在统一的跨领域基准中实例化完整的分类体系，涵盖具有代表性的优化器、模型规模以及从语言模型预训练到图像分类的训练机制，系统分析每个方法家族在多个效果目标上的表现，并阐明其权衡关系。因此，OmniOpt为研究社区提供了在明确机制和目标假设下选择优化器的操作坐标系，并为优化器社区的未来发展方向绘制了蓝图。

English

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.