FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models

Xiaoan Ding, Kevin Gimpel


Abstract
Variational autoencoders (VAEs) are widely used for latent variable modeling of text. We focus on variations that learn expressive prior distributions over the latent variable. We find that existing training strategies are not effective for learning rich priors, so we propose adding the importance-sampled log marginal likelihood as a second term to the standard VAE objective to help when learning the prior. Doing so improves results for all priors evaluated, including a novel choice for sentence VAEs based on normalizing flows (NF). Priors parameterized with NF are no longer constrained to a specific distribution family, allowing a more flexible way to encode the data distribution. Our model, which we call FlowPrior, shows a substantial improvement in language modeling tasks compared to strong baselines. We demonstrate that FlowPrior learns an expressive prior with analysis and several forms of evaluation involving generation.
Anthology ID:
2021.naacl-main.259
Volume:
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
June
Year:
2021
Address:
Online
Editors:
Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, Yichao Zhou
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3242–3258
Language:
URL:
https://aclanthology.org/2021.naacl-main.259
DOI:
10.18653/v1/2021.naacl-main.259
Bibkey:
Cite (ACL):
Xiaoan Ding and Kevin Gimpel. 2021. FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3242–3258, Online. Association for Computational Linguistics.
Cite (Informal):
FlowPrior: Learning Expressive Priors for Latent Variable Sentence Models (Ding & Gimpel, NAACL 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.naacl-main.259.pdf
Video:
 https://aclanthology.org/2021.naacl-main.259.mp4
Data
SNLI