The strange geometry of skip-gram with negative sampling

David Mimno, Laure Thompson


Abstract
Despite their ubiquity, word embeddings trained with skip-gram negative sampling (SGNS) remain poorly understood. We find that vector positions are not simply determined by semantic similarity, but rather occupy a narrow cone, diametrically opposed to the context vectors. We show that this geometric concentration depends on the ratio of positive to negative examples, and that it is neither theoretically nor empirically inherent in related embedding algorithms.
Anthology ID:
D17-1308
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
2873–2878
Language:
URL:
https://aclanthology.org/D17-1308
DOI:
10.18653/v1/D17-1308
Bibkey:
Cite (ACL):
David Mimno and Laure Thompson. 2017. The strange geometry of skip-gram with negative sampling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2873–2878, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
The strange geometry of skip-gram with negative sampling (Mimno & Thompson, EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1308.pdf
Video:
 https://aclanthology.org/D17-1308.mp4