<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chen-etal-2023-locality">
    <titleInfo>
        <title>The Locality and Symmetry of Positional Encodings</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Lihu</namePart>
        <namePart type="family">Chen</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Gael</namePart>
        <namePart type="family">Varoquaux</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Fabian</namePart>
        <namePart type="family">Suchanek</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given recent findings that various positional encodings are insensitive to word order. In this work, we conduct a systematic study of positional encodings in Bidirectional Masked Language Models (BERT-style) , which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We show that the two properties are closely correlated with the performances of downstream tasks; (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results are the basis for developing better PEs for transformer-based language models.</abstract>
    <identifier type="citekey">chen-etal-2023-locality</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.955</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.955</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>14313</start>
            <end>14331</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T The Locality and Symmetry of Positional Encodings
%A Chen, Lihu
%A Varoquaux, Gael
%A Suchanek, Fabian
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F chen-etal-2023-locality
%X Positional Encodings (PEs) are used to inject word-order information into transformer-based language models. While they can significantly enhance the quality of sentence representations, their specific contribution to language models is not fully understood, especially given recent findings that various positional encodings are insensitive to word order. In this work, we conduct a systematic study of positional encodings in Bidirectional Masked Language Models (BERT-style) , which complements existing work in three aspects: (1) We uncover the core function of PEs by identifying two common properties, Locality and Symmetry; (2) We show that the two properties are closely correlated with the performances of downstream tasks; (3) We quantify the weakness of current PEs by introducing two new probing tasks, on which current PEs perform poorly. We believe that these results are the basis for developing better PEs for transformer-based language models.
%R 10.18653/v1/2023.findings-emnlp.955
%U https://aclanthology.org/2023.findings-emnlp.955
%U https://doi.org/10.18653/v1/2023.findings-emnlp.955
%P 14313-14331

Markdown (Informal)

[The Locality and Symmetry of Positional Encodings](https://aclanthology.org/2023.findings-emnlp.955) (Chen et al., Findings 2023)

• The Locality and Symmetry of Positional Encodings (Chen et al., Findings 2023)

ACL

• Lihu Chen, Gael Varoquaux, and Fabian Suchanek. 2023. The Locality and Symmetry of Positional Encodings. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14313–14331, Singapore. Association for Computational Linguistics.