@inproceedings{zhao-etal-2026-towards,
title = "Towards a Comprehensive Scaling Law of Mixture-of-Experts",
author = "Zhao, Guoliang and
Fu, Yuhan and
Li, Shuaipeng and
Sun, Samm and
Xie, Ruobing and
Wang, An and
Han, Weidong and
Yang, Zhen and
Sun, Weixuan and
Zhang, Yudong and
Xu, Cheng-zhong and
Wang, Di and
Jiang, Jie",
editor = "Lee, Hung-Yi and
Dev, Sunipa and
Li, Jessy and
Elliott, Desmond",
booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
month = oct,
year = "2026",
address = "Budapest, Hungary",
publisher = "Association for Computational Linguistics",
url = "https://aclanthology.org/2026.emnlp-main.1227/",
pages = "27994--28022",
ISBN = "979-8-89176-459-0",
abstract = "Mixture-of-Experts (MoE) has emerged as the dominant paradigm for parameter-efficient scaling and cost-effective deployment of Large Language Models (LLMs). However, scaling laws developed for dense models are fundamentally inapplicable to MoE architectures, owing to three critical challenges: the multiplicity of governing factors, their intricate interdependencies, and the non-monotonic nature of their effects on model performance. These challenges collectively demand a rigorous and fine-grained investigation into MoE-specific scaling laws. In this work, we present a systematic decomposition of MoE configurations, identifying five key factors that influence model performance from both size and structural perspectives: data size ($D$), total model size ($N$), activated model size ($N_a$), number of active experts ($G$), and ratio of shared experts ($S$). We conduct 450 controlled experiments to characterize the marginal effects of each factor, culminating in a comprehensive and precise joint MoE scaling law. Building on this foundation, we derive the theoretically optimal and practically efficiency-aware optimal configurations for $G$, $S$ and $N_a/N$ with detailed analyses. Our findings reveal that the optimal settings for $G$ and $S$ are invariant to model and data size, while the optimal activation ratio $N_a/N$ grows sparser as $N$ increases. Our proposed MoE scaling law provides accurate and insightful guidance for future MoE model design and training."
}<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="zhao-etal-2026-towards">
<titleInfo>
<title>Towards a Comprehensive Scaling Law of Mixture-of-Experts</title>
</titleInfo>
<name type="personal">
<namePart type="given">Guoliang</namePart>
<namePart type="family">Zhao</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yuhan</namePart>
<namePart type="family">Fu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Shuaipeng</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Samm</namePart>
<namePart type="family">Sun</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Ruobing</namePart>
<namePart type="family">Xie</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">An</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Weidong</namePart>
<namePart type="family">Han</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Zhen</namePart>
<namePart type="family">Yang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Weixuan</namePart>
<namePart type="family">Sun</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Yudong</namePart>
<namePart type="family">Zhang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Cheng-zhong</namePart>
<namePart type="family">Xu</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Di</namePart>
<namePart type="family">Wang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jie</namePart>
<namePart type="family">Jiang</namePart>
<role>
<roleTerm authority="marcrelator" type="text">author</roleTerm>
</role>
</name>
<originInfo>
<dateIssued>2026-10</dateIssued>
</originInfo>
<typeOfResource>text</typeOfResource>
<relatedItem type="host">
<titleInfo>
<title>Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing</title>
</titleInfo>
<name type="personal">
<namePart type="given">Hung-Yi</namePart>
<namePart type="family">Lee</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Sunipa</namePart>
<namePart type="family">Dev</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Jessy</namePart>
<namePart type="family">Li</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<name type="personal">
<namePart type="given">Desmond</namePart>
<namePart type="family">Elliott</namePart>
<role>
<roleTerm authority="marcrelator" type="text">editor</roleTerm>
</role>
</name>
<originInfo>
<publisher>Association for Computational Linguistics</publisher>
<place>
<placeTerm type="text">Budapest, Hungary</placeTerm>
</place>
</originInfo>
<genre authority="marcgt">conference publication</genre>
<identifier type="isbn">979-8-89176-459-0</identifier>
</relatedItem>
<abstract>Mixture-of-Experts (MoE) has emerged as the dominant paradigm for parameter-efficient scaling and cost-effective deployment of Large Language Models (LLMs). However, scaling laws developed for dense models are fundamentally inapplicable to MoE architectures, owing to three critical challenges: the multiplicity of governing factors, their intricate interdependencies, and the non-monotonic nature of their effects on model performance. These challenges collectively demand a rigorous and fine-grained investigation into MoE-specific scaling laws. In this work, we present a systematic decomposition of MoE configurations, identifying five key factors that influence model performance from both size and structural perspectives: data size (D), total model size (N), activated model size (N_a), number of active experts (G), and ratio of shared experts (S). We conduct 450 controlled experiments to characterize the marginal effects of each factor, culminating in a comprehensive and precise joint MoE scaling law. Building on this foundation, we derive the theoretically optimal and practically efficiency-aware optimal configurations for G, S and N_a/N with detailed analyses. Our findings reveal that the optimal settings for G and S are invariant to model and data size, while the optimal activation ratio N_a/N grows sparser as N increases. Our proposed MoE scaling law provides accurate and insightful guidance for future MoE model design and training.</abstract>
<identifier type="citekey">zhao-etal-2026-towards</identifier>
<location>
<url>https://aclanthology.org/2026.emnlp-main.1227/</url>
</location>
<part>
<date>2026-10</date>
<extent unit="page">
<start>27994</start>
<end>28022</end>
</extent>
</part>
</mods>
</modsCollection>
%0 Conference Proceedings
%T Towards a Comprehensive Scaling Law of Mixture-of-Experts
%A Zhao, Guoliang
%A Fu, Yuhan
%A Li, Shuaipeng
%A Sun, Samm
%A Xie, Ruobing
%A Wang, An
%A Han, Weidong
%A Yang, Zhen
%A Sun, Weixuan
%A Zhang, Yudong
%A Xu, Cheng-zhong
%A Wang, Di
%A Jiang, Jie
%Y Lee, Hung-Yi
%Y Dev, Sunipa
%Y Li, Jessy
%Y Elliott, Desmond
%S Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing
%D 2026
%8 October
%I Association for Computational Linguistics
%C Budapest, Hungary
%@ 979-8-89176-459-0
%F zhao-etal-2026-towards
%X Mixture-of-Experts (MoE) has emerged as the dominant paradigm for parameter-efficient scaling and cost-effective deployment of Large Language Models (LLMs). However, scaling laws developed for dense models are fundamentally inapplicable to MoE architectures, owing to three critical challenges: the multiplicity of governing factors, their intricate interdependencies, and the non-monotonic nature of their effects on model performance. These challenges collectively demand a rigorous and fine-grained investigation into MoE-specific scaling laws. In this work, we present a systematic decomposition of MoE configurations, identifying five key factors that influence model performance from both size and structural perspectives: data size (D), total model size (N), activated model size (N_a), number of active experts (G), and ratio of shared experts (S). We conduct 450 controlled experiments to characterize the marginal effects of each factor, culminating in a comprehensive and precise joint MoE scaling law. Building on this foundation, we derive the theoretically optimal and practically efficiency-aware optimal configurations for G, S and N_a/N with detailed analyses. Our findings reveal that the optimal settings for G and S are invariant to model and data size, while the optimal activation ratio N_a/N grows sparser as N increases. Our proposed MoE scaling law provides accurate and insightful guidance for future MoE model design and training.
%U https://aclanthology.org/2026.emnlp-main.1227/
%P 27994-28022
Markdown (Informal)
[Towards a Comprehensive Scaling Law of Mixture-of-Experts](https://aclanthology.org/2026.emnlp-main.1227/) (Zhao et al., EMNLP 2026)
ACL
- Guoliang Zhao, Yuhan Fu, Shuaipeng Li, Samm Sun, Ruobing Xie, An Wang, Weidong Han, Zhen Yang, Weixuan Sun, Yudong Zhang, Cheng-zhong Xu, Di Wang, and Jie Jiang. 2026. Towards a Comprehensive Scaling Law of Mixture-of-Experts. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, pages 27994–28022, Budapest, Hungary. Association for Computational Linguistics.