Yupeng Su
Author directory2026
Muon2: Boosting Muon via Adaptive Second-Moment Preconditioning
Ziyue Liu | Ruijie Zhang | Zhengyang Wang | Yequan Zhao | Yupeng Su | Zi Yang | Zheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Ziyue Liu | Ruijie Zhang | Zhengyang Wang | Yequan Zhao | Yupeng Su | Zi Yang | Zheng Zhang
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonalization. However, the orthogonalization quality of Muon hinges on the number of Newton–Schulz (NS) iterations performed, which poses efficiency challenges due to its non-trivial computation and communication cost. We propose Muon2, an extension of Muon, to improve both quality and efficiency by applying Adam-style adaptive second-moment preconditioning before orthogonalization. Our key insight is that the core challenge of polar approximation in Muon lies in the ill-conditioned momentum matrix, of which the spectrum is substantially improved by Muon2, leading to faster convergence toward a practically sufficient orthogonalization. We further characterize the practical orthogonalization quality via directional alignment, under which Muon2 demonstrates dramatic improvement over Muon at each polar step. Across GPT, LLaMA, and Mixture-of-Experts pre-training experiments up to 13B parameters, Muon2 (and its memory-efficient variant Muon2-F that preserves most of its benefits) consistently outperforms Muon and its variants while reducing NS iterations by 40%, and saves up to 1/4 training time over Muon when achieving the same loss.