Zhimeng Zhao
Author directory2026
Making Every Verified Token Count: Adaptive Verification for MoE Speculative Decoding
Lehan Pan | Ziyang Tao | Xiao Wang | Zhimeng Zhao | Meng Wang | Yanyong Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Lehan Pan | Ziyang Tao | Xiao Wang | Zhimeng Zhao | Meng Wang | Yanyong Zhang
Findings of the Association for Computational Linguistics: EMNLP 2026
Tree-based speculative decoding accelerates autoregressive generation by verifying multiple draft candidates in parallel, but this advantage weakens for sparse Mixture-of-Experts (MoE) models. As the draft tree grows, different branches activate different experts, expanding the union of activated experts and substantially increasing target-side verification cost. We propose EVICT, a training-free, hyperparameter-free, and lossless adaptive verification method for MoE speculative decoding. EVICT makes every verified token count by truncating the draft tree before target verification and retaining only the cost-effective prefix. It leverages fine-grained drafter signals to estimate candidate benefit, combines them with offline-profiled verification cost, and remains highly compatible with the high-performance graph-based serving framework SGLang. Extensive experiments on diverse MoE backbones and benchmarks show that EVICT achieves up to 2.35× speedup over autoregressive decoding and an average 1.21× speedup over the state-of-the-art baseline EAGLE-3, while significantly reducing unnecessary expert activations during verification.