Geonho Lee
Author directory2026
OPERA-MoE: Output-aware Routing Perturbation Alignment via Observable Discrepancy for MoE Quantization
Geonho Lee | Hancheol Park | Tae-Ho Kim | Jungwook Choi
Findings of the Association for Computational Linguistics: EMNLP 2026
Geonho Lee | Hancheol Park | Tae-Ho Kim | Jungwook Choi
Findings of the Association for Computational Linguistics: EMNLP 2026
Mixture-of-Experts (MoE) architectures have become a dominant paradigm for scaling large language models (LLMs), but their large memory footprint still makes post-training quantization (PTQ) essential for practical deployment. Unlike dense LLMs, quantized MoE models exhibit routing perturbations, where quantization alters expert routing behavior. Existing router alignment methods mitigate this issue by uniformly minimizing routing discrepancies across all tokens. However, we observe that routing perturbations exhibit highly heterogeneous relationships with observable output discrepancy: many tokens preserve nearly identical output distributions despite substantial routing changes. Motivated by this observation, we propose OPERA, an output-aware router alignment strategy for quantized MoE models that selectively prioritizes tokens exhibiting observable output discrepancy after quantization. We further analyze output-preserving routing perturbations and show that they frequently occur on highly predictable tokens, where substantial routing perturbations and hidden-state deviations may not strongly affect projected output distributions. Experiments across multiple MoE LLMs demonstrate that OPERA generally improves over conventional router alignment, with particularly noticeable gains under aggressive low-bit quantization settings.
Boundary-Aware Expert Ordering Preservation with Next-Router Supervision for MoE Quantization
Hancheol Park | Geonho Lee | Tae-Ho Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Hancheol Park | Geonho Lee | Tae-Ho Kim
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Mixture-of-Experts (MoE) language models scale model capacity through sparse expert activation, but their large parameter footprint remains costly to deploy. Post-training quantization (PTQ) offers a practical compression strategy; however, quantization errors can alter router decisions and cause the quantized model to activate experts different from those selected by its full-precision counterpart. Prior router-aware PTQ objectives mitigate such expert-shift by aligning router outputs or preserving top-k expert orderings and margins at a given router. Yet sequential block-wise PTQ introduces an additional failure mode: quantizing a decoder block changes the representations received by the next MoE router, potentially causing downstream expert-shift that is not directly optimized by existing objectives. To address this issue, we propose MENDS-MoE (Margin-Preserving Expert Orderings with Next-Router Decision Supervision), an auxiliary calibration objective that supervises current-block quantization using routing decisions at the next MoE router. MENDS-MoE further introduces a boundary-aware margin loss that preserves orderings among selected experts while maintaining their separation from unselected experts near the top-k boundary. It is applied only during calibration and adds no inference-time computation. Across three MoE language models under 4-bit and 3-bit-average weight-only quantization, MENDS-MoE achieves the best average downstream accuracy and the lowest perplexity in most evaluated model–bit settings. Ablation studies further support the contributions of both next-router supervision and boundary-aware margin preservation.