<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="saadi-etal-2022-comparative">
    <titleInfo>
        <title>Comparative Analysis of Cross-lingual Contextualized Word Embeddings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Hossain</namePart>
        <namePart type="given">Shaikh</namePart>
        <namePart type="family">Saadi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Viktor</namePart>
        <namePart type="family">Hangya</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Tobias</namePart>
        <namePart type="family">Eder</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexander</namePart>
        <namePart type="family">Fraser</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2nd Workshop on Multi-lingual Representation Learning (MRL)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Duygu</namePart>
            <namePart type="family">Ataman</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Hila</namePart>
            <namePart type="family">Gonen</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sebastian</namePart>
            <namePart type="family">Ruder</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Orhan</namePart>
            <namePart type="family">Firat</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Gözde</namePart>
            <namePart type="family">Gül Sahin</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Jamshidbek</namePart>
            <namePart type="family">Mirzakhalov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Abu Dhabi, United Arab Emirates (Hybrid)</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Contextualized word embeddings have emerged as the most important tool for performing NLP tasks in a large variety of languages. In order to improve the cross- lingual representation and transfer learning quality, contextualized embedding alignment techniques, such as mapping and model fine-tuning, are employed. Existing techniques however are time-, data- and computational resource-intensive. In this paper we analyze these techniques by utilizing three tasks: bilingual lexicon induction (BLI), word retrieval and cross-lingual natural language inference (XNLI) for a high resource (German-English) and a low resource (Bengali-English) language pair. In contrast to previous works which focus only on a few popular models, we compare five multilingual and seven monolingual language models and investigate the effect of various aspects on their performance, such as vocabulary size, number of languages used for training and number of parameters. Additionally, we propose a parameter-, data- and runtime-efficient technique which can be trained with 10% of the data, less than 10% of the time and have less than 5% of the trainable parameters compared to model fine-tuning. We show that our proposed method is competitive with resource heavy models, even outperforming them in some cases, even though it relies on less resource</abstract>
    <identifier type="citekey">saadi-etal-2022-comparative</identifier>
    <identifier type="doi">10.18653/v1/2022.mrl-1.6</identifier>
    <location>
        <url>https://aclanthology.org/2022.mrl-1.6</url>
    </location>
    <part>
        <date>2022-12</date>
        <extent unit="page">
            <start>64</start>
            <end>75</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Comparative Analysis of Cross-lingual Contextualized Word Embeddings
%A Saadi, Hossain Shaikh
%A Hangya, Viktor
%A Eder, Tobias
%A Fraser, Alexander
%Y Ataman, Duygu
%Y Gonen, Hila
%Y Ruder, Sebastian
%Y Firat, Orhan
%Y Gül Sahin, Gözde
%Y Mirzakhalov, Jamshidbek
%S Proceedings of the 2nd Workshop on Multi-lingual Representation Learning (MRL)
%D 2022
%8 December
%I Association for Computational Linguistics
%C Abu Dhabi, United Arab Emirates (Hybrid)
%F saadi-etal-2022-comparative
%X Contextualized word embeddings have emerged as the most important tool for performing NLP tasks in a large variety of languages. In order to improve the cross- lingual representation and transfer learning quality, contextualized embedding alignment techniques, such as mapping and model fine-tuning, are employed. Existing techniques however are time-, data- and computational resource-intensive. In this paper we analyze these techniques by utilizing three tasks: bilingual lexicon induction (BLI), word retrieval and cross-lingual natural language inference (XNLI) for a high resource (German-English) and a low resource (Bengali-English) language pair. In contrast to previous works which focus only on a few popular models, we compare five multilingual and seven monolingual language models and investigate the effect of various aspects on their performance, such as vocabulary size, number of languages used for training and number of parameters. Additionally, we propose a parameter-, data- and runtime-efficient technique which can be trained with 10% of the data, less than 10% of the time and have less than 5% of the trainable parameters compared to model fine-tuning. We show that our proposed method is competitive with resource heavy models, even outperforming them in some cases, even though it relies on less resource
%R 10.18653/v1/2022.mrl-1.6
%U https://aclanthology.org/2022.mrl-1.6
%U https://doi.org/10.18653/v1/2022.mrl-1.6
%P 64-75

Markdown (Informal)

[Comparative Analysis of Cross-lingual Contextualized Word Embeddings](https://aclanthology.org/2022.mrl-1.6) (Saadi et al., MRL 2022)

• Comparative Analysis of Cross-lingual Contextualized Word Embeddings (Saadi et al., MRL 2022)

ACL

• Hossain Shaikh Saadi, Viktor Hangya, Tobias Eder, and Alexander Fraser. 2022. Comparative Analysis of Cross-lingual Contextualized Word Embeddings. In Proceedings of the 2nd Workshop on Multi-lingual Representation Learning (MRL), pages 64–75, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.