Ren Lianhai
Author directory2026
Compressing Only What Is Compressible: Memory-Efficient Multiscale Muon via Selective Low-Rank Projection
Ren Lianhai | Yeyun Gong
Findings of the Association for Computational Linguistics: EMNLP 2026
Ren Lianhai | Yeyun Gong
Findings of the Association for Computational Linguistics: EMNLP 2026
Multiscale Muon maintains K momentum EMAs at different decay rates 𝛽k, doubling or tripling the optimizer-state memory of Muon. We show that the EMA update is a first-order low-pass filter and Multiscale Muon is therefore a filter bank on the gradient stream, with each band tuned to a different temporal scale. Under this view, the empirical fact that Multiscale Muon outperforms any single-𝛽 filter is a direct consequence of multi-band coverage; and a sharp, asymmetric compressibility property follows from the bandwidth—the narrow-band (high-𝛽) buffer is dominated by the slow, low-rank component of the gradient and is safe to project, while the wide-band (low-𝛽) buffer carries high-frequency content whose effective rank is large and cannot be compressed without hurting convergence. We formalize this asymmetry, instantiate it in SMuG (Selective Multiscale Muon with GaLore) by applying GaLore only to high-𝛽 buffers, and prove an algebraic impossibility result ruling out any non-negative linear spectral score from predicting the band-mixing coefficient 𝜆⋆ in closed form. On 200,M GPT-2 pretraining (n=4 seeds), SMuG (K=2, r/n=0.125) reaches PPL 21.66±0.05, statistically tied with full-rank Multiscale Muon (21.69±0.03), at 1.25× Muon’s optimizer memory (including the persistent GaLore basis V) versus 2.00× for full Multiscale Muon—recovering 75% of the added overhead at no average quality cost.