Biplav Srivastava


2019

pdf bib
SWOW-8500: Word Association task for Intrinsic Evaluation of Word Embeddings
Avijit Thawani | Biplav Srivastava | Anil Singh
Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP

Downstream evaluation of pretrained word embeddings is expensive, more so for tasks where current state of the art models are very large architectures. Intrinsic evaluation using word similarity or analogy datasets, on the other hand, suffers from several disadvantages. We propose a novel intrinsic evaluation task employing large word association datasets (particularly the Small World of Words dataset). We observe correlations not just between performances on SWOW-8500 and previously proposed intrinsic tasks of word similarity prediction, but also with downstream tasks (eg. Text Classification and Natural Language Inference). Most importantly, we report better confidence intervals for scores on our word association task, with no fall in correlation with downstream performance.