<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dikkala-etal-2023-benefits">
    <titleInfo>
        <title>On the Benefits of Learning to Route in Mixture-of-Experts Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Nishanth</namePart>
        <namePart type="family">Dikkala</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikhil</namePart>
        <namePart type="family">Ghosh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Raghu</namePart>
        <namePart type="family">Meka</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Rina</namePart>
        <namePart type="family">Panigrahy</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Nikhil</namePart>
        <namePart type="family">Vyas</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Xin</namePart>
        <namePart type="family">Wang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Mixture-of-Expert (MoE) Transformer models, such as the Switch Transformer, allow us to successfully scale up model sizes while keeping the amount of compute time fixed. Prior work has established the computational efficiency benefits of using these models. A core component of these models is a router that routes input tokens to different experts in a layer. We show theoretical and empirical evidence that the router’s ability to route tokens intelligently confers a significant advantage to MoE models. We study synthetic settings where the input data is distributed in clusters and show theoretically and empirically that the router learns to route the inputs according to these clusters. Then we perform experiments on real data using the T5X library, where we observe that a trainable router confers a non-trivial benefit instead of a non-trainable router.</abstract>
    <identifier type="citekey">dikkala-etal-2023-benefits</identifier>
    <identifier type="doi">10.18653/v1/2023.emnlp-main.583</identifier>
    <location>
        <url>https://aclanthology.org/2023.emnlp-main.583</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>9376</start>
            <end>9396</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On the Benefits of Learning to Route in Mixture-of-Experts Models
%A Dikkala, Nishanth
%A Ghosh, Nikhil
%A Meka, Raghu
%A Panigrahy, Rina
%A Vyas, Nikhil
%A Wang, Xin
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F dikkala-etal-2023-benefits
%X Mixture-of-Expert (MoE) Transformer models, such as the Switch Transformer, allow us to successfully scale up model sizes while keeping the amount of compute time fixed. Prior work has established the computational efficiency benefits of using these models. A core component of these models is a router that routes input tokens to different experts in a layer. We show theoretical and empirical evidence that the router’s ability to route tokens intelligently confers a significant advantage to MoE models. We study synthetic settings where the input data is distributed in clusters and show theoretically and empirically that the router learns to route the inputs according to these clusters. Then we perform experiments on real data using the T5X library, where we observe that a trainable router confers a non-trivial benefit instead of a non-trainable router.
%R 10.18653/v1/2023.emnlp-main.583
%U https://aclanthology.org/2023.emnlp-main.583
%U https://doi.org/10.18653/v1/2023.emnlp-main.583
%P 9376-9396

Markdown (Informal)

[On the Benefits of Learning to Route in Mixture-of-Experts Models](https://aclanthology.org/2023.emnlp-main.583) (Dikkala et al., EMNLP 2023)

• On the Benefits of Learning to Route in Mixture-of-Experts Models (Dikkala et al., EMNLP 2023)

ACL

• Nishanth Dikkala, Nikhil Ghosh, Raghu Meka, Rina Panigrahy, Nikhil Vyas, and Xin Wang. 2023. On the Benefits of Learning to Route in Mixture-of-Experts Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9376–9396, Singapore. Association for Computational Linguistics.