What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP

Oskar Wysocki, Malina Florea, Dónal Landers, André Freitas


Abstract
SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.
Anthology ID:
2021.eval4nlp-1.21
Volume:
Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venues:
EMNLP | Eval4NLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
209–229
Language:
URL:
https://aclanthology.org/2021.eval4nlp-1.21
DOI:
10.18653/v1/2021.eval4nlp-1.21
Bibkey:
Cite (ACL):
Oskar Wysocki, Malina Florea, Dónal Landers, and André Freitas. 2021. What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP. In Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems, pages 209–229, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP (Wysocki et al., Eval4NLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.eval4nlp-1.21.pdf