Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics

Zhe Zhao, Tao Liu, Shen Li, Bofang Li, Xiaoyong Du


Abstract
The existing word representation methods mostly limit their information source to word co-occurrence statistics. In this paper, we introduce ngrams into four representation methods: SGNS, GloVe, PPMI matrix, and its SVD factorization. Comprehensive experiments are conducted on word analogy and similarity tasks. The results show that improved word representations are learned from ngram co-occurrence statistics. We also demonstrate that the trained ngram representations are useful in many aspects such as finding antonyms and collocations. Besides, a novel approach of building co-occurrence matrix is proposed to alleviate the hardware burdens brought by ngrams.
Anthology ID:
D17-1023
Erratum e1:
D17-1023e1
Volume:
Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Editors:
Martha Palmer, Rebecca Hwa, Sebastian Riedel
Venue:
EMNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
244–253
Language:
URL:
https://aclanthology.org/D17-1023
DOI:
10.18653/v1/D17-1023
Bibkey:
Cite (ACL):
Zhe Zhao, Tao Liu, Shen Li, Bofang Li, and Xiaoyong Du. 2017. Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 244–253, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics (Zhao et al., EMNLP 2017)
Copy Citation:
PDF:
https://aclanthology.org/D17-1023.pdf