Pion: 一种基于正交等价变换的谱保持优化器

Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation

May 12, 2026

作者: Kexuan Shi, Hanxuan Li, Zeju Qiu, Yandong Wen, Simon Buchholz, Weiyang Liu

cs.AI

摘要

我们提出Pion，一种基于正交等价变换、用于大语言模型（LLM）训练的谱保持优化器。与Adam和Muon等加法优化器不同，Pion通过左、右正交变换更新每个权重矩阵，从而在训练过程中保持其奇异值不变。这产生了一种优化机制，能够在固定权重矩阵谱范数的同时，调节其几何结构。我们推导了Pion的更新规则，系统检验了其设计选择，并分析了其收敛行为及若干关键特性。实验结果表明，在LLM预训练和微调中，Pion均提供了稳定且具有竞争力的替代方案，可与标准优化器相媲美。

English

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.