<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="ai-fang-2022-leveraging">
    <titleInfo>
        <title>Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Xi</namePart>
        <namePart type="family">Ai</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Bin</namePart>
        <namePart type="family">Fang</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2022-05</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Smaranda</namePart>
            <namePart type="family">Muresan</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Preslav</namePart>
            <namePart type="family">Nakov</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Aline</namePart>
            <namePart type="family">Villavicencio</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Dublin, Ireland</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Transformer and UT (universal transformer), all tokens are equally processed towards depth. Inspired by the equilibrium phenomenon, we present a lazy transition, a mechanism to adjust the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is encouraged to oscillate around a relaxed equilibrium. Our experiments show that LT outperforms baseline models on several tasks of machine translation, pre-training, Learning to Execute, and LAMBADA.</abstract>
    <identifier type="citekey">ai-fang-2022-leveraging</identifier>
    <identifier type="doi">10.18653/v1/2022.acl-long.208</identifier>
    <location>
        <url>https://aclanthology.org/2022.acl-long.208</url>
    </location>
    <part>
        <date>2022-05</date>
        <extent unit="page">
            <start>2904</start>
            <end>2924</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling
%A Ai, Xi
%A Fang, Bin
%Y Muresan, Smaranda
%Y Nakov, Preslav
%Y Villavicencio, Aline
%S Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
%D 2022
%8 May
%I Association for Computational Linguistics
%C Dublin, Ireland
%F ai-fang-2022-leveraging
%X In sequence modeling, certain tokens are usually less ambiguous than others, and representations of these tokens require fewer refinements for disambiguation. However, given the nature of attention-based models like Transformer and UT (universal transformer), all tokens are equally processed towards depth. Inspired by the equilibrium phenomenon, we present a lazy transition, a mechanism to adjust the significance of iterative refinements for each token representation. Our lazy transition is deployed on top of UT to build LT (lazy transformer), where all tokens are processed unequally towards depth. Eventually, LT is encouraged to oscillate around a relaxed equilibrium. Our experiments show that LT outperforms baseline models on several tasks of machine translation, pre-training, Learning to Execute, and LAMBADA.
%R 10.18653/v1/2022.acl-long.208
%U https://aclanthology.org/2022.acl-long.208
%U https://doi.org/10.18653/v1/2022.acl-long.208
%P 2904-2924

Markdown (Informal)

[Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling](https://aclanthology.org/2022.acl-long.208) (Ai & Fang, ACL 2022)

• Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling (Ai & Fang, ACL 2022)

ACL

• Xi Ai and Bin Fang. 2022. Leveraging Relaxed Equilibrium by Lazy Transition for Sequence Modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2904–2924, Dublin, Ireland. Association for Computational Linguistics.