Ren Lianhai

Author directory

2026

Multiscale Muon maintains K momentum EMAs at different decay rates 𝛽k, doubling or tripling the optimizer-state memory of Muon. We show that the EMA update is a first-order low-pass filter and Multiscale Muon is therefore a filter bank on the gradient stream, with each band tuned to a different temporal scale. Under this view, the empirical fact that Multiscale Muon outperforms any single-𝛽 filter is a direct consequence of multi-band coverage; and a sharp, asymmetric compressibility property follows from the bandwidth—the narrow-band (high-𝛽) buffer is dominated by the slow, low-rank component of the gradient and is safe to project, while the wide-band (low-𝛽) buffer carries high-frequency content whose effective rank is large and cannot be compressed without hurting convergence. We formalize this asymmetry, instantiate it in SMuG (Selective Multiscale Muon with GaLore) by applying GaLore only to high-𝛽 buffers, and prove an algebraic impossibility result ruling out any non-negative linear spectral score from predicting the band-mixing coefficient 𝜆⋆ in closed form. On 200,M GPT-2 pretraining (n=4 seeds), SMuG (K=2, r/n=0.125) reaches PPL 21.66±0.05, statistically tied with full-rank Multiscale Muon (21.69±0.03), at 1.25× Muon’s optimizer memory (including the persistent GaLore basis V) versus 2.00× for full Multiscale Muon—recovering 75% of the added overhead at no average quality cost.