One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations

Jesper Brink Andersen, Mikkel Bak Bertelsen, Mikkel Hørby Schou, Manuel R. Ciosici, Ira Assent


Abstract
Word embeddings are an active topic in the NLP research community. State-of-the-art neural models achieve high performance on downstream tasks, albeit at the cost of computationally expensive training. Cost aware solutions require cheaper models that still achieve good performance. We present several reproduction studies of intrinsic evaluation tasks that evaluate non-contextual word representations in multiple languages. Furthermore, we present 50-8-8, a new data set for the outlier identification task, which avoids limitations of the original data set, such as ambiguous words, infrequent words, and multi-word tokens, while increasing the number of test cases. The data set is expanded to contain semantic and syntactic tests and is multilingual (English, German, and Italian). We provide an in-depth analysis of word embedding models with a range of hyper-parameters. Our analysis shows the suitability of different models and hyper-parameters for different tasks and the greater difficulty of representing German and Italian languages.
Anthology ID:
2020.eval4nlp-1.12
Volume:
Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2020
Address:
Online
Editors:
Steffen Eger, Yang Gao, Maxime Peyrard, Wei Zhao, Eduard Hovy
Venue:
Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
120–130
Language:
URL:
https://aclanthology.org/2020.eval4nlp-1.12
DOI:
10.18653/v1/2020.eval4nlp-1.12
Bibkey:
Cite (ACL):
Jesper Brink Andersen, Mikkel Bak Bertelsen, Mikkel Hørby Schou, Manuel R. Ciosici, and Ira Assent. 2020. One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 120–130, Online. Association for Computational Linguistics.
Cite (Informal):
One of these words is not like the other: a reproduction of outlier identification using non-contextual word representations (Brink Andersen et al., Eval4NLP 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.eval4nlp-1.12.pdf
Optional supplementary material:
 2020.eval4nlp-1.12.OptionalSupplementaryMaterial.pdf
Video:
 https://slideslive.com/38939717
Code
 jesperbrink/50-8-8
Data
Penn Treebank