Subword-based Compact Reconstruction of Word Embeddings

Shota Sasaki, Jun Suzuki, Kentaro Inui


Abstract
The idea of subword-based word embeddings has been proposed in the literature, mainly for solving the out-of-vocabulary (OOV) word problem observed in standard word-based word embeddings. In this paper, we propose a method of reconstructing pre-trained word embeddings using subword information that can effectively represent a large number of subword embeddings in a considerably small fixed space. The key techniques of our method are twofold: memory-shared embeddings and a variant of the key-value-query self-attention mechanism. Our experiments show that our reconstructed subword-based embeddings can successfully imitate well-trained word embeddings in a small fixed space while preventing quality degradation across several linguistic benchmark datasets, and can simultaneously predict effective embeddings of OOV words. We also demonstrate the effectiveness of our reconstruction method when we apply them to downstream tasks.
Anthology ID:
N19-1353
Volume:
Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)
Month:
June
Year:
2019
Address:
Minneapolis, Minnesota
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3498–3508
Language:
URL:
https://aclanthology.org/N19-1353
DOI:
10.18653/v1/N19-1353
Bibkey:
Cite (ACL):
Shota Sasaki, Jun Suzuki, and Kentaro Inui. 2019. Subword-based Compact Reconstruction of Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3498–3508, Minneapolis, Minnesota. Association for Computational Linguistics.
Cite (Informal):
Subword-based Compact Reconstruction of Word Embeddings (Sasaki et al., NAACL 2019)
Copy Citation:
PDF:
https://aclanthology.org/N19-1353.pdf
Supplementary:
 N19-1353.Supplementary.pdf
Video:
 https://vimeo.com/347430247
Code
 losyer/compact_reconstruction
Data
CoNLL-2003SNLI