<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="kulikov-etal-2021-mode">
    <titleInfo>
        <title>Mode recovery in neural autoregressive sequence modeling</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Ilia</namePart>
        <namePart type="family">Kulikov</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sean</namePart>
        <namePart type="family">Welleck</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Kyunghyun</namePart>
        <namePart type="family">Cho</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2021-08</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021)</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Zornitsa</namePart>
            <namePart type="family">Kozareva</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Sujith</namePart>
            <namePart type="family">Ravi</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Andreas</namePart>
            <namePart type="family">Vlachos</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Priyanka</namePart>
            <namePart type="family">Agrawal</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">André</namePart>
            <namePart type="family">Martins</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Despite its wide use, recent studies have revealed unexpected and undesirable properties of neural autoregressive sequence models trained with maximum likelihood, such as an unreasonably high affinity to short sequences after training and to infinitely long sequences at decoding time. We propose to study these phenomena by investigating how the modes, or local maxima, of a distribution are maintained throughout the full learning chain of the ground-truth, empirical, learned and decoding-induced distributions, via the newly proposed mode recovery cost. We design a tractable testbed where we build three types of ground-truth distributions: (1) an LSTM based structured distribution, (2) an unstructured distribution where probability of a sequence does not depend on its content, and (3) a product of these two which we call a semi-structured distribution. Our study reveals both expected and unexpected findings. First, starting with data collection, mode recovery cost strongly relies on the ground-truth distribution and is most costly with the semi-structured distribution. Second, after learning, mode recovery cost from the ground-truth distribution may increase or decrease compared to data collection, with the largest cost degradation occurring with the semi-structured ground-truth distribution. Finally, the ability of the decoding-induced distribution to recover modes from the learned distribution is highly impacted by the choices made earlier in the learning chain. We conclude that future research must consider the entire learning chain in order to fully understand the potentials and perils and to further improve neural autoregressive sequence models.</abstract>
    <identifier type="citekey">kulikov-etal-2021-mode</identifier>
    <identifier type="doi">10.18653/v1/2021.spnlp-1.5</identifier>
    <location>
        <url>https://aclanthology.org/2021.spnlp-1.5</url>
    </location>
    <part>
        <date>2021-08</date>
        <extent unit="page">
            <start>44</start>
            <end>52</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Mode recovery in neural autoregressive sequence modeling
%A Kulikov, Ilia
%A Welleck, Sean
%A Cho, Kyunghyun
%Y Kozareva, Zornitsa
%Y Ravi, Sujith
%Y Vlachos, Andreas
%Y Agrawal, Priyanka
%Y Martins, André
%S Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021)
%D 2021
%8 August
%I Association for Computational Linguistics
%C Online
%F kulikov-etal-2021-mode
%X Despite its wide use, recent studies have revealed unexpected and undesirable properties of neural autoregressive sequence models trained with maximum likelihood, such as an unreasonably high affinity to short sequences after training and to infinitely long sequences at decoding time. We propose to study these phenomena by investigating how the modes, or local maxima, of a distribution are maintained throughout the full learning chain of the ground-truth, empirical, learned and decoding-induced distributions, via the newly proposed mode recovery cost. We design a tractable testbed where we build three types of ground-truth distributions: (1) an LSTM based structured distribution, (2) an unstructured distribution where probability of a sequence does not depend on its content, and (3) a product of these two which we call a semi-structured distribution. Our study reveals both expected and unexpected findings. First, starting with data collection, mode recovery cost strongly relies on the ground-truth distribution and is most costly with the semi-structured distribution. Second, after learning, mode recovery cost from the ground-truth distribution may increase or decrease compared to data collection, with the largest cost degradation occurring with the semi-structured ground-truth distribution. Finally, the ability of the decoding-induced distribution to recover modes from the learned distribution is highly impacted by the choices made earlier in the learning chain. We conclude that future research must consider the entire learning chain in order to fully understand the potentials and perils and to further improve neural autoregressive sequence models.
%R 10.18653/v1/2021.spnlp-1.5
%U https://aclanthology.org/2021.spnlp-1.5
%U https://doi.org/10.18653/v1/2021.spnlp-1.5
%P 44-52

Markdown (Informal)

[Mode recovery in neural autoregressive sequence modeling](https://aclanthology.org/2021.spnlp-1.5) (Kulikov et al., spnlp 2021)

• Mode recovery in neural autoregressive sequence modeling (Kulikov et al., spnlp 2021)

ACL

• Ilia Kulikov, Sean Welleck, and Kyunghyun Cho. 2021. Mode recovery in neural autoregressive sequence modeling. In Proceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), pages 44–52, Online. Association for Computational Linguistics.