CroaTPAS: A Survey-based Evaluation

Costanza Marini


Abstract
The Croatian Typed Predicate Argument Structures resource is a Croatian/English bilingual digital dictionary of corpus-derived verb valency structures, whose argument slots have been annotated with Semantic Types labels following the CPA methodology. CroaTPAS is tailor-made to represent verb polysemy and currently contains 180 Croatian verbs for a total of 683 different verbs senses. In order to evaluate the resource both in terms of identified Croatian verb senses, as well as of the English descriptions explaining them, an online survey based on a multiple-choice sense disambiguation task was devised, pilot tested and distributed among respondents following a snowball sampling methodology. Answers from 30 respondents were collected and compared against a yardstick set of answers in line with CroaTPAS’s sense distinctions. Jaccard similarity index was used as a measure of agreement. Since the multiple-choice items respondents answered to were based on a representative selection of CroaTPAS verbs, they allowed for a generalization of the results to the whole of the resource.
Anthology ID:
2022.isa-1.10
Volume:
Proceedings of the 18th Joint ACL - ISO Workshop on Interoperable Semantic Annotation within LREC2022
Month:
June
Year:
2022
Address:
Marseille, France
Editor:
Harry Bunt
Venue:
ISA
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
76–80
Language:
URL:
https://aclanthology.org/2022.isa-1.10
DOI:
Bibkey:
Cite (ACL):
Costanza Marini. 2022. CroaTPAS: A Survey-based Evaluation. In Proceedings of the 18th Joint ACL - ISO Workshop on Interoperable Semantic Annotation within LREC2022, pages 76–80, Marseille, France. European Language Resources Association.
Cite (Informal):
CroaTPAS: A Survey-based Evaluation (Marini, ISA 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.isa-1.10.pdf