On Importance Sampling-Based Evaluation of Latent Language Models

Robert L. Logan IV, Matt Gardner, Sameer Singh


Abstract
Language models that use additional latent structures (e.g., syntax trees, coreference chains, knowledge graph links) provide several advantages over traditional language models. However, likelihood-based evaluation of these models is often intractable as it requires marginalizing over the latent space. Existing works avoid this issue by using importance sampling. Although this approach has asymptotic guarantees, analysis is rarely conducted on the effect of decisions such as sample size and choice of proposal distribution on the reported estimates. In this paper, we carry out this analysis for three models: RNNG, EntityNLM, and KGLM. In addition, we elucidate subtle differences in how importance sampling is applied in these works that can have substantial effects on the final estimates, as well as provide theoretical results which reinforce the validity of this technique.
Anthology ID:
2020.acl-main.196
Volume:
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
Month:
July
Year:
2020
Address:
Online
Editors:
Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2171–2176
Language:
URL:
https://aclanthology.org/2020.acl-main.196
DOI:
10.18653/v1/2020.acl-main.196
Bibkey:
Cite (ACL):
Robert L. Logan IV, Matt Gardner, and Sameer Singh. 2020. On Importance Sampling-Based Evaluation of Latent Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2171–2176, Online. Association for Computational Linguistics.
Cite (Informal):
On Importance Sampling-Based Evaluation of Latent Language Models (Logan IV et al., ACL 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.acl-main.196.pdf
Video:
 http://slideslive.com/38929173