<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="dufter-etal-2022-position">
    <titleInfo>
        <title>Position Information in Transformers: An Overview</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Philipp</namePart>
        <namePart type="family">Dufter</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Martin</namePart>
        <namePart type="family">Schmitt</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Hinrich</namePart>
        <namePart type="family">Schütze</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-09</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <genre authority="bibutilsgt">journal article</genre>
    <relatedItem type="host">
        <titleInfo>
            <title>Computational Linguistics</title>
        </titleInfo>
        <originInfo>
            <issuance>continuing</issuance>
            <publisher>MIT Press</publisher>
            <place>
                <placeTerm type="text">Cambridge, MA</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">periodical</genre>
        <genre authority="bibutilsgt">academic journal</genre>
    </relatedItem>
    <abstract>Transformers are arguably the main workhorse in recent natural language processing research. By definition, a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; and (4) provide stimuli for future research.</abstract>
    <identifier type="citekey">dufter-etal-2022-position</identifier>
    <identifier type="doi">10.1162/coli_a_00445</identifier>
    <location>
        <url>https://aclanthology.org/2022.cl-3.7</url>
    </location>
    <part>
        <date>2022-09</date>
        <detail type="volume"><number>48</number></detail>
        <detail type="issue"><number>3</number></detail>
        <extent unit="page">
            <start>733</start>
            <end>763</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Journal Article
%T Position Information in Transformers: An Overview
%A Dufter, Philipp
%A Schmitt, Martin
%A Schütze, Hinrich
%J Computational Linguistics
%D 2022
%8 September
%V 48
%N 3
%I MIT Press
%C Cambridge, MA
%F dufter-etal-2022-position
%X Transformers are arguably the main workhorse in recent natural language processing research. By definition, a Transformer is invariant with respect to reordering of the input. However, language is inherently sequential and word order is essential to the semantics and syntax of an utterance. In this article, we provide an overview and theoretical comparison of existing methods to incorporate position information into Transformer models. The objectives of this survey are to (1) showcase that position information in Transformer is a vibrant and extensive research area; (2) enable the reader to compare existing methods by providing a unified notation and systematization of different approaches along important model dimensions; (3) indicate what characteristics of an application should be taken into account when selecting a position encoding; and (4) provide stimuli for future research.
%R 10.1162/coli_a_00445
%U https://aclanthology.org/2022.cl-3.7
%U https://doi.org/10.1162/coli_a_00445
%P 733-763

Markdown (Informal)

[Position Information in Transformers: An Overview](https://aclanthology.org/2022.cl-3.7) (Dufter et al., CL 2022)

• Position Information in Transformers: An Overview (Dufter et al., CL 2022)

ACL

• Philipp Dufter, Martin Schmitt, and Hinrich Schütze. 2022. Position Information in Transformers: An Overview. Computational Linguistics, 48(3):733–763.