<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="chi-etal-2023-transformer">
    <titleInfo>
        <title>Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ta-Chung</namePart>
        <namePart type="family">Chi</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Ting-Han</namePart>
        <namePart type="family">Fan</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Alexander</namePart>
        <namePart type="family">Rudnicky</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Peter</namePart>
        <namePart type="family">Ramadge</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2023-12</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Findings of the Association for Computational Linguistics: EMNLP 2023</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Houda</namePart>
            <namePart type="family">Bouamor</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Juan</namePart>
            <namePart type="family">Pino</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Kalika</namePart>
            <namePart type="family">Bali</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Singapore</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.</abstract>
    <identifier type="citekey">chi-etal-2023-transformer</identifier>
    <identifier type="doi">10.18653/v1/2023.findings-emnlp.397</identifier>
    <location>
        <url>https://aclanthology.org/2023.findings-emnlp.397</url>
    </location>
    <part>
        <date>2023-12</date>
        <extent unit="page">
            <start>5972</start>
            <end>5984</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation
%A Chi, Ta-Chung
%A Fan, Ting-Han
%A Rudnicky, Alexander
%A Ramadge, Peter
%Y Bouamor, Houda
%Y Pino, Juan
%Y Bali, Kalika
%S Findings of the Association for Computational Linguistics: EMNLP 2023
%D 2023
%8 December
%I Association for Computational Linguistics
%C Singapore
%F chi-etal-2023-transformer
%X Unlike recurrent models, conventional wisdom has it that Transformers cannot perfectly model regular languages. Inspired by the notion of working memory, we propose a new Transformer variant named RegularGPT. With its novel combination of Weight-Sharing, Adaptive-Depth, and Sliding-Dilated-Attention, RegularGPT constructs working memory along the depth dimension, thereby enabling efficient and successful modeling of regular languages such as PARITY. We further test RegularGPT on the task of natural language length extrapolation and surprisingly find that it rediscovers the local windowed attention effect deemed necessary in prior work for length extrapolation.
%R 10.18653/v1/2023.findings-emnlp.397
%U https://aclanthology.org/2023.findings-emnlp.397
%U https://doi.org/10.18653/v1/2023.findings-emnlp.397
%P 5972-5984

Markdown (Informal)

[Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation](https://aclanthology.org/2023.findings-emnlp.397) (Chi et al., Findings 2023)

• Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation (Chi et al., Findings 2023)

ACL

• Ta-Chung Chi, Ting-Han Fan, Alexander Rudnicky, and Peter Ramadge. 2023. Transformer Working Memory Enables Regular Language Reasoning And Natural Language Length Extrapolation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5972–5984, Singapore. Association for Computational Linguistics.