Fast and Discriminative Semantic Embedding

Rob Koopman, Shenghui Wang, Gwenn Englebienne


Abstract
The embedding of words and documents in compact, semantically meaningful vector spaces is a crucial part of modern information systems. Deep Learning models are powerful but their hyperparameter selection is often complex and they are expensive to train, and while pre-trained models are available, embeddings trained on general corpora are not necessarily well-suited to domain specific tasks. We propose a novel embedding method which extends random projection by weighting and projecting raw term embeddings orthogonally to an average language vector, thus improving the discriminating power of resulting term embeddings, and build more meaningful document embeddings by assigning appropriate weights to individual terms. We describe how updating the term embeddings online as we process the training data results in an extremely efficient method, in terms of both computational and memory requirements. Our experiments show highly competitive results with various state-of-the-art embedding methods on different tasks, including the standard STS benchmark and a subject prediction task, at a fraction of the computational cost.
Anthology ID:
W19-0420
Volume:
Proceedings of the 13th International Conference on Computational Semantics - Long Papers
Month:
May
Year:
2019
Address:
Gothenburg, Sweden
Editors:
Simon Dobnik, Stergios Chatzikyriakidis, Vera Demberg
Venue:
IWCS
SIG:
SIGSEM
Publisher:
Association for Computational Linguistics
Note:
Pages:
235–246
Language:
URL:
https://aclanthology.org/W19-0420
DOI:
10.18653/v1/W19-0420
Bibkey:
Cite (ACL):
Rob Koopman, Shenghui Wang, and Gwenn Englebienne. 2019. Fast and Discriminative Semantic Embedding. In Proceedings of the 13th International Conference on Computational Semantics - Long Papers, pages 235–246, Gothenburg, Sweden. Association for Computational Linguistics.
Cite (Informal):
Fast and Discriminative Semantic Embedding (Koopman et al., IWCS 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-0420.pdf