SuperSim: a test set for word similarity and relatedness in Swedish

Simon Hengchen, Nina Tahmasebi


Abstract
Language models are notoriously difficult to evaluate. We release SuperSim, a large-scale similarity and relatedness test set for Swedish built with expert human judgements. The test set is composed of 1,360 word-pairs independently judged for both relatedness and similarity by five annotators. We evaluate three different models (Word2Vec, fastText, and GloVe) trained on two separate Swedish datasets, namely the Swedish Gigaword corpus and a Swedish Wikipedia dump, to provide a baseline for future comparison. We will release the fully annotated test set, code, models, and data.
Anthology ID:
2021.nodalida-main.27
Volume:
Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)
Month:
May 31--2 June
Year:
2021
Address:
Reykjavik, Iceland (Online)
Venue:
NoDaLiDa
SIG:
Publisher:
Linköping University Electronic Press, Sweden
Note:
Pages:
268–275
Language:
URL:
https://aclanthology.org/2021.nodalida-main.27
DOI:
Bibkey:
Cite (ACL):
Simon Hengchen and Nina Tahmasebi. 2021. SuperSim: a test set for word similarity and relatedness in Swedish. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 268–275, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
Cite (Informal):
SuperSim: a test set for word similarity and relatedness in Swedish (Hengchen & Tahmasebi, NoDaLiDa 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.nodalida-main.27.pdf