SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge

Roberto Zamparelli, Shammur Chowdhury, Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta, Md. Arid Hasan, Giulia Venturi


Abstract
We report the results of the SemEval 2022 Task 3, PreTENS, on evaluation the acceptability of simple sentences containing constructions whose two arguments are presupposed to be or not to be in an ordered taxonomic relation. The task featured two sub-tasks articulated as: (i) binary prediction task and (ii) regression task, predicting the acceptability in a continuous scale. The sentences were artificially generated in three languages (English, Italian and French). 21 systems, with 8 system papers were submitted for the task, all based on various types of fine-tuned transformer systems, often with ensemble methods and various data augmentation techniques. The best systems reached an F1-macro score of 94.49 (sub-task1) and a Spearman correlation coefficient of 0.80 (sub-task2), with interesting variations in specific constructions and/or languages.
Anthology ID:
2022.semeval-1.29
Volume:
Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022)
Month:
July
Year:
2022
Address:
Seattle, United States
Editors:
Guy Emerson, Natalie Schluter, Gabriel Stanovsky, Ritesh Kumar, Alexis Palmer, Nathan Schneider, Siddharth Singh, Shyam Ratan
Venue:
SemEval
SIG:
SIGLEX
Publisher:
Association for Computational Linguistics
Note:
Pages:
228–238
Language:
URL:
https://aclanthology.org/2022.semeval-1.29
DOI:
10.18653/v1/2022.semeval-1.29
Bibkey:
Cite (ACL):
Roberto Zamparelli, Shammur Chowdhury, Dominique Brunato, Cristiano Chesi, Felice Dell’Orletta, Md. Arid Hasan, and Giulia Venturi. 2022. SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 228–238, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
SemEval-2022 Task 3: PreTENS-Evaluating Neural Networks on Presuppositional Semantic Knowledge (Zamparelli et al., SemEval 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.semeval-1.29.pdf
Video:
 https://aclanthology.org/2022.semeval-1.29.mp4