<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rajaee-monz-2024-analyzing">
    <titleInfo>
        <title>Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sara</namePart>
        <namePart type="family">Rajaee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Christof</namePart>
        <namePart type="family">Monz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2024-03</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Yvette</namePart>
            <namePart type="family">Graham</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Matthew</namePart>
            <namePart type="family">Purver</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">St. Julian’s, Malta</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we question to what extent the current evaluation benchmarks and setups accurately measure zero-shot cross-lingual knowledge transfer. In this work, we challenge the assumption that high zero-shot performance on target tasks reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. Through extensive experiments and analysis, we show that the observed high performance of multilingual models can be largely attributed to factors not requiring the transfer of actual linguistic knowledge, such as task- and surface-level knowledge. More specifically, we observe what has been transferred across languages is mostly data artifacts and biases, especially for low-resource languages. Our findings highlight the overlooked drawbacks of existing cross-lingual test data and evaluation setups, calling for a more nuanced understanding of the cross-lingual capabilities of multilingual models.</abstract>
    <identifier type="citekey">rajaee-monz-2024-analyzing</identifier>
    <location>
        <url>https://aclanthology.org/2024.eacl-long.177</url>
    </location>
    <part>
        <date>2024-03</date>
        <extent unit="page">
            <start>2895</start>
            <end>2914</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models
%A Rajaee, Sara
%A Monz, Christof
%Y Graham, Yvette
%Y Purver, Matthew
%S Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2024
%8 March
%I Association for Computational Linguistics
%C St. Julian’s, Malta
%F rajaee-monz-2024-analyzing
%X Recent advances in training multilingual language models on large datasets seem to have shown promising results in knowledge transfer across languages and achieve high performance on downstream tasks. However, we question to what extent the current evaluation benchmarks and setups accurately measure zero-shot cross-lingual knowledge transfer. In this work, we challenge the assumption that high zero-shot performance on target tasks reflects high cross-lingual ability by introducing more challenging setups involving instances with multiple languages. Through extensive experiments and analysis, we show that the observed high performance of multilingual models can be largely attributed to factors not requiring the transfer of actual linguistic knowledge, such as task- and surface-level knowledge. More specifically, we observe what has been transferred across languages is mostly data artifacts and biases, especially for low-resource languages. Our findings highlight the overlooked drawbacks of existing cross-lingual test data and evaluation setups, calling for a more nuanced understanding of the cross-lingual capabilities of multilingual models.
%U https://aclanthology.org/2024.eacl-long.177
%P 2895-2914

Markdown (Informal)

[Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models](https://aclanthology.org/2024.eacl-long.177) (Rajaee & Monz, EACL 2024)

• Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models (Rajaee & Monz, EACL 2024)

ACL

• Sara Rajaee and Christof Monz. 2024. Analyzing the Evaluation of Cross-Lingual Knowledge Transfer in Multilingual Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2895–2914, St. Julian’s, Malta. Association for Computational Linguistics.