Using semi-experts to derive judgments on word sense alignment: a pilot study

Soojeong Eom, Markus Dickinson, Graham Katz


Abstract
The overall goal of this project is to evaluate the performance of word sense alignment (WSA) systems, focusing on obtaining examples appropriate to language learners. Building a gold standard dataset based on human expert judgments is costly in time and labor, and thus we gauge the utility of using semi-experts in performing the annotation. In an online survey, we present a sense of a target word from one dictionary with senses from the other dictionary, asking for judgments of relatedness. We note the difficulty of agreement, yet the utility in using such results to evaluate WSA work. We find that one's treatment of related senses heavily impacts the results for WSA.
Anthology ID:
L12-1380
Volume:
Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
Month:
May
Year:
2012
Address:
Istanbul, Turkey
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Mehmet Uğur Doğan, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
605–611
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/652_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Soojeong Eom, Markus Dickinson, and Graham Katz. 2012. Using semi-experts to derive judgments on word sense alignment: a pilot study. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 605–611, Istanbul, Turkey. European Language Resources Association (ELRA).
Cite (Informal):
Using semi-experts to derive judgments on word sense alignment: a pilot study (Eom et al., LREC 2012)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2012/pdf/652_Paper.pdf