<?xml version="1.0" encoding="UTF-8"?>
<modsCollection xmlns="http://www.loc.gov/mods/v3">
<mods ID="logan-iv-etal-2020-importance">
    <titleInfo>
        <title>On Importance Sampling-Based Evaluation of Latent Language Models</title>
    </titleInfo>
    <name type="personal">
        <namePart type="given">Robert</namePart>
        <namePart type="given">L</namePart>
        <namePart type="family">Logan IV</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Matt</namePart>
        <namePart type="family">Gardner</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <name type="personal">
        <namePart type="given">Sameer</namePart>
        <namePart type="family">Singh</namePart>
        <role>
            <roleTerm authority="marcrelator" type="text">author</roleTerm>
        </role>
    </name>
    <originInfo>
        <dateIssued>2020-07</dateIssued>
    </originInfo>
    <typeOfResource>text</typeOfResource>
    <relatedItem type="host">
        <titleInfo>
            <title>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</title>
        </titleInfo>
        <name type="personal">
            <namePart type="given">Dan</namePart>
            <namePart type="family">Jurafsky</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joyce</namePart>
            <namePart type="family">Chai</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Natalie</namePart>
            <namePart type="family">Schluter</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <name type="personal">
            <namePart type="given">Joel</namePart>
            <namePart type="family">Tetreault</namePart>
            <role>
                <roleTerm authority="marcrelator" type="text">editor</roleTerm>
            </role>
        </name>
        <originInfo>
            <publisher>Association for Computational Linguistics</publisher>
            <place>
                <placeTerm type="text">Online</placeTerm>
            </place>
        </originInfo>
        <genre authority="marcgt">conference publication</genre>
    </relatedItem>
    <abstract>Language models that use additional latent structures (e.g., syntax trees, coreference chains, knowledge graph links) provide several advantages over traditional language models. However, likelihood-based evaluation of these models is often intractable as it requires marginalizing over the latent space. Existing works avoid this issue by using importance sampling. Although this approach has asymptotic guarantees, analysis is rarely conducted on the effect of decisions such as sample size and choice of proposal distribution on the reported estimates. In this paper, we carry out this analysis for three models: RNNG, EntityNLM, and KGLM. In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.</abstract>
    <identifier type="citekey">logan-iv-etal-2020-importance</identifier>
    <identifier type="doi">10.18653/v1/2020.acl-main.196</identifier>
    <location>
        <url>https://aclanthology.org/2020.acl-main.196/</url>
    </location>
    <part>
        <date>2020-07</date>
        <extent unit="page">
            <start>2171</start>
            <end>2176</end>
        </extent>
    </part>
</mods>
</modsCollection>

%0 Conference Proceedings
%T On Importance Sampling-Based Evaluation of Latent Language Models
%A Logan IV, Robert L.
%A Gardner, Matt
%A Singh, Sameer
%Y Jurafsky, Dan
%Y Chai, Joyce
%Y Schluter, Natalie
%Y Tetreault, Joel
%S Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
%D 2020
%8 July
%I Association for Computational Linguistics
%C Online
%F logan-iv-etal-2020-importance
%X Language models that use additional latent structures (e.g., syntax trees, coreference chains, knowledge graph links) provide several advantages over traditional language models. However, likelihood-based evaluation of these models is often intractable as it requires marginalizing over the latent space. Existing works avoid this issue by using importance sampling. Although this approach has asymptotic guarantees, analysis is rarely conducted on the effect of decisions such as sample size and choice of proposal distribution on the reported estimates. In this paper, we carry out this analysis for three models: RNNG, EntityNLM, and KGLM. In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.
%R 10.18653/v1/2020.acl-main.196
%U https://aclanthology.org/2020.acl-main.196/
%U https://doi.org/10.18653/v1/2020.acl-main.196
%P 2171-2176

Markdown (Informal)

[On Importance Sampling-Based Evaluation of Latent Language Models](https://aclanthology.org/2020.acl-main.196/) (Logan IV et al., ACL 2020)

• On Importance Sampling-Based Evaluation of Latent Language Models (Logan IV et al., ACL 2020)

ACL

• Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2020. On Importance Sampling-Based Evaluation of Latent Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2171–2176, Online. Association for Computational Linguistics.