<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="lee-hwang-2023-multilingual">
    <titleInfo>
        <title>Multilingual Lottery Tickets to Pretrain Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Jaeseong</namePart>
        <namePart type="family">Lee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Seung-won</namePart>
        <namePart type="family">Hwang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a per-language subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scale-then-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zero-shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.</abstract>
    <identifier type="citekey">lee-hwang-2023-multilingual</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.629</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.629</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>9387</start>
            <end>9398</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Multilingual Lottery Tickets to Pretrain Language Models
%A Lee, Jaeseong
%A Hwang, Seung-won
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F lee-hwang-2023-multilingual
%X The curse of multilinguality in training multilingual pretrained language models (mPLMs) refers to the negative interference between languages, especially when the capacity is limited. While increasing the capacity may appear intuitive for overcoming this curse, it negatively affects both training and inference costs. Our distinction is pursuing the competing goals of reducing negative interference, while keeping capacity per each language more or less the same. Specifically, we first scale the model to reduce interference, then search for a per-language subnetwork, or a lottery ticket, with comparable performance to the full model. According to lottery ticket hypothesis, this scale-then-find-ticket approach alleviates interfering signals as in the scaled model, but redistributes parameters to keep the parameters reduced. Finally, to avoid the cost of multiple retraining for searching multilingual tickets, we explore zero-shot neural architecture search (NAS) methods. We investigate the most appropriate zero-shot NAS method to find multilingual tickets. Our proposed multilingual tickets reduce the inference cost of models for each languages, while boosting the performances. The ticket search cost is negligible and tickets found qualitatively preserve linguistic similarity. Our code is publicly available.
%R 10.18653/v1/2023.findings-emnlp.629
%U https://aclanthology.org/2023.findings-emnlp.629
%U https://doi.org/10.18653/v1/2023.findings-emnlp.629
%P 9387-9398

Markdown (Informal)

[Multilingual Lottery Tickets to Pretrain Language Models](https://aclanthology.org/2023.findings-emnlp.629) (Lee & Hwang, Findings 2023)

• Multilingual Lottery Tickets to Pretrain Language Models (Lee & Hwang, Findings 2023)

ACL

• Jaeseong Lee and Seung-won Hwang. 2023. Multilingual Lottery Tickets to Pretrain Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9387–9398, Singapore. Association for Computational Linguistics.