Tang Hongwei

Author directory

2026

Mixture-of-Experts (MoE) models route each token to a subset of expert networks, increasing capacity while keeping per-token computation sparse. Many deployed MoEs use a fixed number of active experts across layers and tasks, despite evidence that layer roles, expert redundancy, and required capacity vary with depth and input difficulty. Existing layer-wise methods usually determine allocations offline and reuse them across tasks, while token-level methods rely on local routing signals without task-level context. We propose MetaNet, a support-set controller that predicts an expert-retention threshold and a bounded routing bias for each layer while keeping the backbone, experts, and original router frozen. On DeepSeek-MoE-16B-Chat, MetaNet provides a tunable accuracy–expert-activation trade-off. Relative to fixed k=6, a conservative setting activates 3.61 experts on average (40% fewer) with comparable MMLU accuracy (0.489 versus 0.474); an aggressive setting activates 2.28 experts on average (62% fewer) at 0.438 accuracy. Trained on MMLU, MetaNet transfers to C-Eval without retraining, activating 2.90 experts on average (52% fewer than fixed k=6) at 0.386 accuracy.