Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors

Katsuhito Sudoh, Kosuke Takahashi, Satoshi Nakamura


Abstract
This paper discusses a classification-based approach to machine translation evaluation, as opposed to a common regression-based approach in the WMT Metrics task. Recent machine translation usually works well but sometimes makes critical errors due to just a few wrong word choices. Our classification-based approach focuses on such errors using several error type labels, for practical machine translation evaluation in an age of neural machine translation. We made additional annotations on the WMT 2015-2017 Metrics datasets with fluency and adequacy labels to distinguish different types of translation errors from syntactic and semantic viewpoints. We present our human evaluation criteria for the corpus development and automatic evaluation experiments using the corpus. The human evaluation corpus will be publicly available upon publication.
Anthology ID:
2021.humeval-1.5
Volume:
Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
Month:
April
Year:
2021
Address:
Online
Editors:
Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, Anastasia Shimorina
Venue:
HumEval
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
46–55
Language:
URL:
https://aclanthology.org/2021.humeval-1.5
DOI:
Bibkey:
Cite (ACL):
Katsuhito Sudoh, Kosuke Takahashi, and Satoshi Nakamura. 2021. Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 46–55, Online. Association for Computational Linguistics.
Cite (Informal):
Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors (Sudoh et al., HumEval 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.humeval-1.5.pdf
Dataset:
 2021.humeval-1.5.Dataset.zip
Video:
 https://www.youtube.com/watch?v=myG72lA2hpo
Video:
 https://aclanthology.org/2021.humeval-1.5.mp4