Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models Siqi Wang author Zhengyu Chen author Bei Li author Keqing He author Min Zhang author Jingang Wang author 2024-11 text Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing Yaser Al-Onaizan editor Mohit Bansal editor Yun-Nung Chen editor Association for Computational Linguistics Miami, Florida, USA conference publication wang-etal-2024-scaling 10.18653/v1/2024.emnlp-main.319 https://aclanthology.org/2024.emnlp-main.319/ 2024-11 5583 5595