EquiformerV3：可扩展、高表达力且通用的SE(3)等变图注意力Transformer模型

EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

April 10, 2026

作者: Yi-Lun Liao, Alexander J. Hoffman, Sabrina C. Shen, Alexandre Duval, Sam Walton Norwood, Tess Smidt

cs.AI

摘要

随着SE(3)等变图神经网络成为三维原子建模的核心工具，提升其效率、表达能力和物理一致性已成为大规模应用的核心挑战。本文推出第三代SE(3)等变图注意力Transformer——EquiformerV3，该模型在效率、表达能力与通用性三个维度实现同步突破。基于EquiformerV2的架构，我们实现了三项关键改进：首先，通过软件实现优化获得1.75倍的加速效果；其次，对EquiformerV2进行简洁有效的改进，包括引入等变合并层归一化、优化前馈网络超参数以及采用平滑径向截断的注意力机制；最后，提出SwiGLU-S^2激活函数，通过引入多体相互作用提升理论表达能力，在降低S^2网格采样复杂度的同时保持严格等变性。SwiGLU-S^2激活与平滑截断注意力机制共同实现了对平滑变化势能面（PES）的精确建模，使EquiformerV3可推广至需要能量守恒模拟和PES高阶导数的任务。结合对非平衡结构去噪（DeNS）的辅助训练策略，改进后的EquiformerV3在OC20、OMat24和Matbench Discovery数据集上取得了最先进的性能。

English

As SE(3)-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the SE(3)-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving 1.75times speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-S^2 activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling S^2 grids. Together, SwiGLU-S^2 activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.