<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="abdaoui-etal-2020-load">
    <titleInfo>
        <title>Load What You Need: Smaller Versions of Mutililingual BERT</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Amine</namePart>
        <namePart type="family">Abdaoui</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Camille</namePart>
        <namePart type="family">Pradel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Grégoire</namePart>
        <namePart type="family">Sigel</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Nafise</namePart>
            <namePart type="given">Sadat</namePart>
            <namePart type="family">Moosavi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Angela</namePart>
            <namePart type="family">Fan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Vered</namePart>
            <namePart type="family">Shwartz</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Goran</namePart>
            <namePart type="family">Glavaš</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Shafiq</namePart>
            <namePart type="family">Joty</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Alex</namePart>
            <namePart type="family">Wang</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Thomas</namePart>
            <namePart type="family">Wolf</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to extract smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.</abstract>
    <identifier type="citekey">abdaoui-etal-2020-load</identifier>
    <identifier type="doi">10.18653/v1/2020.sustainlp-1.16</identifier>
    <location>
        <url>https://aclanthology.org/2020.sustainlp-1.16</url>
    </location>
    <part>
        <date>2020-11</date>
        <extent unit="page">
            <start>119</start>
            <end>123</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Load What You Need: Smaller Versions of Mutililingual BERT
%A Abdaoui, Amine
%A Pradel, Camille
%A Sigel, Grégoire
%Y Moosavi, Nafise Sadat
%Y Fan, Angela
%Y Shwartz, Vered
%Y Glavaš, Goran
%Y Joty, Shafiq
%Y Wang, Alex
%Y Wolf, Thomas
%S Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing
%D 2020
%8 November
%I Association for Computational Linguistics
%C Online
%F abdaoui-etal-2020-load
%X Pre-trained Transformer-based models are achieving state-of-the-art results on a variety of Natural Language Processing data sets. However, the size of these models is often a drawback for their deployment in real production applications. In the case of multilingual models, most of the parameters are located in the embeddings layer. Therefore, reducing the vocabulary size should have an important impact on the total number of parameters. In this paper, we propose to extract smaller models that handle fewer number of languages according to the targeted corpora. We present an evaluation of smaller versions of multilingual BERT on the XNLI data set, but we believe that this method may be applied to other multilingual transformers. The obtained results confirm that we can generate smaller models that keep comparable results, while reducing up to 45% of the total number of parameters. We compared our models with DistilmBERT (a distilled version of multilingual BERT) and showed that unlike language reduction, distillation induced a 1.7% to 6% drop in the overall accuracy on the XNLI data set. The presented models and code are publicly available.
%R 10.18653/v1/2020.sustainlp-1.16
%U https://aclanthology.org/2020.sustainlp-1.16
%U https://doi.org/10.18653/v1/2020.sustainlp-1.16
%P 119-123

Markdown (Informal)

[Load What You Need: Smaller Versions of Mutililingual BERT](https://aclanthology.org/2020.sustainlp-1.16) (Abdaoui et al., sustainlp 2020)

• Load What You Need: Smaller Versions of Mutililingual BERT (Abdaoui et al., sustainlp 2020)

ACL

• Amine Abdaoui, Camille Pradel, and Grégoire Sigel. 2020. Load What You Need: Smaller Versions of Mutililingual BERT. In Proceedings of SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, pages 119–123, Online. Association for Computational Linguistics.