Does Summary Evaluation Survive Translation to Other Languages?

Spencer Braun, Oleg Vasilyev, Neslihan Iskender, John Bohannon


Abstract
The creation of a quality summarization dataset is an expensive, time-consuming effort, requiring the production and evaluation of summaries by both trained humans and machines. The returns to such an effort would increase significantly if the dataset could be used in additional languages without repeating human annotations. To investigate how much we can trust machine translation of summarization datasets, we translate the English SummEval dataset to seven languages and compare performances across automatic evaluation measures. We explore equivalence testing as the appropriate statistical paradigm for evaluating correlations between human and automated scoring of summaries. We also consider the effect of translation on the relative performance between measures. We find some potential for dataset reuse in languages similar to the source and along particular dimensions of summary quality. Our code and data can be found at https://github.com/PrimerAI/primer-research/.
Anthology ID:
2022.naacl-main.173
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2425–2435
Language:
URL:
https://aclanthology.org/2022.naacl-main.173
DOI:
10.18653/v1/2022.naacl-main.173
Bibkey:
Cite (ACL):
Spencer Braun, Oleg Vasilyev, Neslihan Iskender, and John Bohannon. 2022. Does Summary Evaluation Survive Translation to Other Languages?. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2425–2435, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Does Summary Evaluation Survive Translation to Other Languages? (Braun et al., NAACL 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.naacl-main.173.pdf