Probabilistic FastText for Multi-Sense Word Embeddings

Ben Athiwaratkun, Andrew Wilson, Anima Anandkumar


Abstract
We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share the “strength” across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, our model is the first to achieve best of both the worlds: multi-sense representations while having enriched semantics on rare words.
Anthology ID:
P18-1001
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Editors:
Iryna Gurevych, Yusuke Miyao
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–11
Language:
URL:
https://aclanthology.org/P18-1001
DOI:
10.18653/v1/P18-1001
Bibkey:
Cite (ACL):
Ben Athiwaratkun, Andrew Wilson, and Anima Anandkumar. 2018. Probabilistic FastText for Multi-Sense Word Embeddings. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1–11, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Probabilistic FastText for Multi-Sense Word Embeddings (Athiwaratkun et al., ACL 2018)
Copy Citation:
PDF:
https://aclanthology.org/P18-1001.pdf
Presentation:
 P18-1001.Presentation.pdf
Video:
 https://aclanthology.org/P18-1001.mp4
Code
 benathi/multisense-prob-fasttext