SubER - A Metric for Automatic Evaluation of Subtitle Quality

Patrick Wilken, Panayota Georgakopoulou, Evgeny Matusov


Abstract
This paper addresses the problem of evaluating the quality of automatically generated subtitles, which includes not only the quality of the machine-transcribed or translated speech, but also the quality of line segmentation and subtitle timing. We propose SubER - a single novel metric based on edit distance with shifts that takes all of these subtitle properties into account. We compare it to existing metrics for evaluating transcription, translation, and subtitle quality. A careful human evaluation in a post-editing scenario shows that the new metric has a high correlation with the post-editing effort and direct human assessment scores, outperforming baseline metrics considering only the subtitle text, such as WER and BLEU, and existing methods to integrate segmentation and timing features.
Anthology ID:
2022.iwslt-1.1
Volume:
Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022)
Month:
May
Year:
2022
Address:
Dublin, Ireland (in-person and online)
Venues:
ACL | IWSLT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–10
Language:
URL:
https://aclanthology.org/2022.iwslt-1.1
DOI:
10.18653/v1/2022.iwslt-1.1
Bibkey:
Cite (ACL):
Patrick Wilken, Panayota Georgakopoulou, and Evgeny Matusov. 2022. SubER - A Metric for Automatic Evaluation of Subtitle Quality. In Proceedings of the 19th International Conference on Spoken Language Translation (IWSLT 2022), pages 1–10, Dublin, Ireland (in-person and online). Association for Computational Linguistics.
Cite (Informal):
SubER - A Metric for Automatic Evaluation of Subtitle Quality (Wilken et al., IWSLT 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.iwslt-1.1.pdf