Multimodal Word Distributions

Ben Athiwaratkun, Andrew Wilson


Abstract
Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective. We show that the resulting approach captures uniquely expressive semantic information, and outperforms alternatives, such as word2vec skip-grams, and Gaussian embeddings, on benchmark datasets such as word similarity and entailment.
Anthology ID:
P17-1151
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Editors:
Regina Barzilay, Min-Yen Kan
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1645–1656
Language:
URL:
https://aclanthology.org/P17-1151
DOI:
10.18653/v1/P17-1151
Bibkey:
Cite (ACL):
Ben Athiwaratkun and Andrew Wilson. 2017. Multimodal Word Distributions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1645–1656, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Multimodal Word Distributions (Athiwaratkun & Wilson, ACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/P17-1151.pdf
Code
 benathi/word2gm +  additional community code