Linguistic Corpus Annotation for Automatic Text Simplification Evaluation

Rémi Cardon, Adrien Bibal, Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Watrin Patrick, Thomas François


Abstract
Evaluating automatic text simplification (ATS) systems is a difficult task that is either performed by automatic metrics or user-based evaluations. However, from a linguistic point-of-view, it is not always clear on what bases these evaluations operate. In this paper, we propose annotations of the ASSET corpus that can be used to shed more light on ATS evaluation. In addition to contributing with this resource, we show how it can be used to analyze SARI’s behavior and to re-evaluate existing ATS systems. We present our insights as a step to improve ATS evaluation protocols in the future.
Anthology ID:
2022.emnlp-main.121
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1842–1866
Language:
URL:
https://aclanthology.org/2022.emnlp-main.121
DOI:
10.18653/v1/2022.emnlp-main.121
Bibkey:
Cite (ACL):
Rémi Cardon, Adrien Bibal, Rodrigo Wilkens, David Alfter, Magali Norré, Adeline Müller, Watrin Patrick, and Thomas François. 2022. Linguistic Corpus Annotation for Automatic Text Simplification Evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1842–1866, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Linguistic Corpus Annotation for Automatic Text Simplification Evaluation (Cardon et al., EMNLP 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.emnlp-main.121.pdf