<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ravishankar-sogaard-2021-impact">
    <titleInfo>
        <title>The Impact of Positional Encodings on Multilingual Compression</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Vinit</namePart>
        <namePart type="family">Ravishankar</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Anders</namePart>
        <namePart type="family">Søgaard</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-11</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing</title>
        </titleInfo>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online and Punta Cana, Dominican Republic</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings. Several modifications have been proposed over the sinusoidal positional encodings used in the original transformer architecture; these include, for instance, separating position encodings and token embeddings, or directly modifying attention weights based on the distance between word pairs. We first show that surprisingly, while these modifications tend to improve monolingual language models, none of them result in better multilingual language models. We then answer why that is: sinusoidal encodings were explicitly designed to facilitate compositionality by allowing linear projections over arbitrary time steps. Higher variances in multilingual training distributions requires higher compression, in which case, compositionality becomes indispensable. Learned absolute positional encodings (e.g., in mBERT) tend to approximate sinusoidal embeddings in multilingual settings, but more complex positional encoding architectures lack the inductive bias to effectively learn cross-lingual alignment. In other words, while sinusoidal positional encodings were designed for monolingual applications, they are particularly useful in multilingual language models.</abstract>
    <identifier type="citekey">ravishankar-sogaard-2021-impact</identifier>
    <identifier type="doi">10.18653/v1/2021.emnlp-main.59</identifier>
    <location>
        <url>https://aclanthology.org/2021.emnlp-main.59</url>
    </location>
    <part>
        <date>2021-11</date>
        <extent unit="page">
            <start>763</start>
            <end>777</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Impact of Positional Encodings on Multilingual Compression
%A Ravishankar, Vinit
%A Søgaard, Anders
%S Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
%D 2021
%8 November
%I Association for Computational Linguistics
%C Online and Punta Cana, Dominican Republic
%F ravishankar-sogaard-2021-impact
%X In order to preserve word-order information in a non-autoregressive setting, transformer architectures tend to include positional knowledge, by (for instance) adding positional encodings to token embeddings. Several modifications have been proposed over the sinusoidal positional encodings used in the original transformer architecture; these include, for instance, separating position encodings and token embeddings, or directly modifying attention weights based on the distance between word pairs. We first show that surprisingly, while these modifications tend to improve monolingual language models, none of them result in better multilingual language models. We then answer why that is: sinusoidal encodings were explicitly designed to facilitate compositionality by allowing linear projections over arbitrary time steps. Higher variances in multilingual training distributions requires higher compression, in which case, compositionality becomes indispensable. Learned absolute positional encodings (e.g., in mBERT) tend to approximate sinusoidal embeddings in multilingual settings, but more complex positional encoding architectures lack the inductive bias to effectively learn cross-lingual alignment. In other words, while sinusoidal positional encodings were designed for monolingual applications, they are particularly useful in multilingual language models.
%R 10.18653/v1/2021.emnlp-main.59
%U https://aclanthology.org/2021.emnlp-main.59
%U https://doi.org/10.18653/v1/2021.emnlp-main.59
%P 763-777

Markdown (Informal)

[The Impact of Positional Encodings on Multilingual Compression](https://aclanthology.org/2021.emnlp-main.59) (Ravishankar & Søgaard, EMNLP 2021)

• The Impact of Positional Encodings on Multilingual Compression (Ravishankar & Søgaard, EMNLP 2021)

ACL

• Vinit Ravishankar and Anders Søgaard. 2021. The Impact of Positional Encodings on Multilingual Compression. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 763–777, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.