Reconstruction of Word Embeddings from Sub-Word Parameters

Karl Stratos


Abstract
Pre-trained word embeddings improve the performance of a neural model at the cost of increasing the model size. We propose to benefit from this resource without paying the cost by operating strictly at the sub-lexical level. Our approach is quite simple: before task-specific training, we first optimize sub-word parameters to reconstruct pre-trained word embeddings using various distance measures. We report interesting results on a variety of tasks: word similarity, word analogy, and part-of-speech tagging.
Anthology ID:
W17-4119
Volume:
Proceedings of the First Workshop on Subword and Character Level Models in NLP
Month:
September
Year:
2017
Address:
Copenhagen, Denmark
Venue:
SCLeM
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
130–135
Language:
URL:
https://aclanthology.org/W17-4119
DOI:
10.18653/v1/W17-4119
Bibkey:
Cite (ACL):
Karl Stratos. 2017. Reconstruction of Word Embeddings from Sub-Word Parameters. In Proceedings of the First Workshop on Subword and Character Level Models in NLP, pages 130–135, Copenhagen, Denmark. Association for Computational Linguistics.
Cite (Informal):
Reconstruction of Word Embeddings from Sub-Word Parameters (Stratos, SCLeM 2017)
Copy Citation:
PDF:
https://aclanthology.org/W17-4119.pdf