<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="pelsmaeker-aziz-2020-effective">
    <titleInfo>
        <title>Effective Estimation of Deep Generative Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Tom</namePart>
        <namePart type="family">Pelsmaeker</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Wilker</namePart>
        <namePart type="family">Aziz</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.</abstract>
    <identifier type="citekey">pelsmaeker-aziz-2020-effective</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.646</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.646</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>7220</start>
            <end>7236</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T Effective Estimation of Deep Generative Language Models
%A Pelsmaeker, Tom
%A Aziz, Wilker
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F pelsmaeker-aziz-2020-effective
%X Advances in variational inference enable parameterisation of probabilistic models by deep neural networks. This combines the statistical transparency of the probabilistic modelling framework with the representational power of deep learning. Yet, due to a problem known as posterior collapse, it is difficult to estimate such models in the context of language modelling effectively. We concentrate on one such model, the variational auto-encoder, which we argue is an important building block in hierarchical probabilistic models of language. This paper contributes a sober view of the problem, a survey of techniques to address it, novel techniques, and extensions to the model. To establish a ranking of techniques, we perform a systematic comparison using Bayesian optimisation and find that many techniques perform reasonably similar, given enough resources. Still, a favourite can be named based on convenience. We also make several empirical observations and recommendations of best practices that should help researchers interested in this exciting field.
%R 10.18653/v1/2020.acl-main.646
%U https://aclanthology.org/2020.acl-main.646
%U https://doi.org/10.18653/v1/2020.acl-main.646
%P 7220-7236

Markdown (Informal)

[Effective Estimation of Deep Generative Language Models](https://aclanthology.org/2020.acl-main.646) (Pelsmaeker & Aziz, ACL 2020)

• Effective Estimation of Deep Generative Language Models (Pelsmaeker & Aziz, ACL 2020)

ACL

• Tom Pelsmaeker and Wilker Aziz. 2020. Effective Estimation of Deep Generative Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7220–7236, Online. Association for Computational Linguistics.