Menghao Li
Author directoryOther people with similar names: Menghao Li
2026
Complexity-Aware Routing: Detecting and Allocating Dynamic Compute for Efficient Mixture-of-Experts Large Language Models
Jiahe Li | Saisai Hu | Menghao Li | Shaolin Zhu | Deyi Xiong
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Jiahe Li | Saisai Hu | Menghao Li | Shaolin Zhu | Deyi Xiong
Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
Current Mixture-of-Experts (MoE) models typically rely on a static Top-k routing mechanism, activating a fixed number of experts for every token regardless of its difficulty. This uniform allocation ignores the inherent variability in linguistic information density that simple functional words require significantly less processing capacity than content-heavy terms. The static routing results in computational waste for easy tokens and potential under-fitting for difficult ones. To address this issue, we propose **CAR**, a Complexity-Aware Routing framework that aligns the computational budget with the intrinsic information density of input tokens. Diverging from previous dynamic methods that rely on unstable heuristics, we ground our routing decision in a dual-signal objective combining Shannon Surprisal and Predictive Entropy to objectively measure token complexity. We implement a decoupled architecture where a Complexity Perception (CP) module utilizes a Recurrent Complexity Memory (RCM) to estimate these signals from a teacher model, guiding a Dynamic Expert Allocator (DEA) to adjust k on the fly. Extensive experiments demonstrate that CAR consistently outperforms both static baselines and SOTA dynamic routing methods. Notably, our approach achieves superior accuracy on reasoning-intensive benchmarks (e.g., boosting MMLU by 0.8 and GSM8K by 1.5 over static routing) while reducing the average number of activated parameters by approximately 10%. Our code is available at https://github.com/Acking646/car.