<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="rajaee-pilehvar-2022-isotropy">
    <titleInfo>
        <title>An Isotropy Analysis in the Multilingual BERT Embedding Space</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Sara</namePart>
        <namePart type="family">Rajaee</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Mohammad</namePart>
        <namePart type="given">Taher</namePart>
        <namePart type="family">Pilehvar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: ACL 2022</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outlier dimensions. We show that, unlike its monolingual counterpart, the multilingual BERT model exhibits no outlier dimension in its representations while it has a highly anisotropic space. There are a few dimensions in the monolingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our analysis indicates that, despite having different degenerated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.</abstract>
    <identifier type="citekey">rajaee-pilehvar-2022-isotropy</identifier>
    <identifier type="doi">10.18653/v1/2022.findings-acl.103</identifier>
    <location>
        <url>https://aclanthology.org/2022.findings-acl.103</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>1309</start>
            <end>1316</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T An Isotropy Analysis in the Multilingual BERT Embedding Space
%A Rajaee, Sara
%A Pilehvar, Mohammad Taher
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Findings of the Association for Computational Linguistics: ACL 2022
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F rajaee-pilehvar-2022-isotropy
%X Several studies have explored various advantages of multilingual pre-trained models (such as multilingual BERT) in capturing shared linguistic knowledge. However, less attention has been paid to their limitations. In this paper, we investigate the multilingual BERT for two known issues of the monolingual models: anisotropic embedding space and outlier dimensions. We show that, unlike its monolingual counterpart, the multilingual BERT model exhibits no outlier dimension in its representations while it has a highly anisotropic space. There are a few dimensions in the monolingual BERT with high contributions to the anisotropic distribution. However, we observe no such dimensions in the multilingual BERT. Furthermore, our experimental results demonstrate that increasing the isotropy of multilingual space can significantly improve its representation power and performance, similarly to what had been observed for monolingual CWRs on semantic similarity tasks. Our analysis indicates that, despite having different degenerated directions, the embedding spaces in various languages tend to be partially similar with respect to their structures.
%R 10.18653/v1/2022.findings-acl.103
%U https://aclanthology.org/2022.findings-acl.103
%U https://doi.org/10.18653/v1/2022.findings-acl.103
%P 1309-1316

Markdown (Informal)

[An Isotropy Analysis in the Multilingual BERT Embedding Space](https://aclanthology.org/2022.findings-acl.103) (Rajaee & Pilehvar, Findings 2022)

• An Isotropy Analysis in the Multilingual BERT Embedding Space (Rajaee & Pilehvar, Findings 2022)

ACL

• Sara Rajaee and Mohammad Taher Pilehvar. 2022. An Isotropy Analysis in the Multilingual BERT Embedding Space. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1309–1316, Dublin, Ireland. Association for Computational Linguistics.