<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="muller-etal-2021-unseen">
    <titleInfo>
        <title>When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Benjamin</namePart>
        <namePart type="family">Muller</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Antonios</namePart>
        <namePart type="family">Anastasopoulos</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Benoît</namePart>
        <namePart type="family">Sagot</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Djamé</namePart>
        <namePart type="family">Seddah</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-06</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly improves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages.</abstract>
    <identifier type="citekey">muller-etal-2021-unseen</identifier>
    <identifier type="doi">10.18653/v1/2021.naacl-main.38</identifier>
    <location>
        <url>https://aclanthology.org/2021.naacl-main.38</url>
    </location>
    <part>
        <date>2021-06</date>
        <extent unit="page">
            <start>448</start>
            <end>462</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models
%A Muller, Benjamin
%A Anastasopoulos, Antonios
%A Sagot, Benoît
%A Seddah, Djamé
%S Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
%D 2021
%8 June
%I Association for Computational Linguistics
%C Online
%F muller-etal-2021-unseen
%X Transfer learning based on pretraining language models on a large amount of raw data has become a new norm to reach state-of-the-art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by comparing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave similarly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to transfer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly improves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages.
%R 10.18653/v1/2021.naacl-main.38
%U https://aclanthology.org/2021.naacl-main.38
%U https://doi.org/10.18653/v1/2021.naacl-main.38
%P 448-462

Markdown (Informal)

[When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models](https://aclanthology.org/2021.naacl-main.38) (Muller et al., NAACL 2021)

• When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models (Muller et al., NAACL 2021)

ACL

• Benjamin Muller, Antonios Anastasopoulos, Benoît Sagot, and Djamé Seddah. 2021. When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462, Online. Association for Computational Linguistics.