Direct vs. indirect evaluation of distributional thesauri

Vincent Claveau, Ewa Kijak


Abstract
With the success of word embedding methods in various Natural Language Processing tasks, all the field of distributional semantics has experienced a renewed interest. Beside the famous word2vec, recent studies have presented efficient techniques to build distributional thesaurus; in particular, Claveau et al. (2014) have already shown that Information Retrieval (IR) tools and concepts can be successfully used to build a thesaurus. In this paper, we address the problem of the evaluation of such thesauri or embedding models and compare their results. Through several experiments and by evaluating directly the results with reference lexicons, we show that the recent IR-based distributional models outperform state-of-the-art systems such as word2vec. Following the work of Claveau and Kijak (2016), we use IR as an applicative framework to indirectly evaluate the generated thesaurus. Here again, this task-based evaluation validates the IR approach used to build the thesaurus. Moreover, it allows us to compare these results with those from the direct evaluation framework used in the literature. The observed differences bring these evaluation habits into question.
Anthology ID:
C16-1173
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Editors:
Yuji Matsumoto, Rashmi Prasad
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
1837–1848
Language:
URL:
https://aclanthology.org/C16-1173
DOI:
Bibkey:
Cite (ACL):
Vincent Claveau and Ewa Kijak. 2016. Direct vs. indirect evaluation of distributional thesauri. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1837–1848, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Direct vs. indirect evaluation of distributional thesauri (Claveau & Kijak, COLING 2016)
Copy Citation:
PDF:
https://aclanthology.org/C16-1173.pdf