Integrating Meaning into Quality Evaluation of Machine Translation

Osman Başkaya, Eray Yildiz, Doruk Tunaoğlu, Mustafa Tolga Eren, A. Seza Doğruöz


Abstract
Machine translation (MT) quality is evaluated through comparisons between MT outputs and the human translations (HT). Traditionally, this evaluation relies on form related features (e.g. lexicon and syntax) and ignores the transfer of meaning reflected in HT outputs. Instead, we evaluate the quality of MT outputs through meaning related features (e.g. polarity, subjectivity) with two experiments. In the first experiment, the meaning related features are compared to human rankings individually. In the second experiment, combinations of meaning related features and other quality metrics are utilized to predict the same human rankings. The results of our experiments confirm the benefit of these features in predicting human evaluation of translation quality in addition to traditional metrics which focus mainly on form.
Anthology ID:
E17-1020
Volume:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Mirella Lapata, Phil Blunsom, Alexander Koller
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
210–219
Language:
URL:
https://aclanthology.org/E17-1020
DOI:
Bibkey:
Cite (ACL):
Osman Başkaya, Eray Yildiz, Doruk Tunaoğlu, Mustafa Tolga Eren, and A. Seza Doğruöz. 2017. Integrating Meaning into Quality Evaluation of Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 210–219, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
Integrating Meaning into Quality Evaluation of Machine Translation (Başkaya et al., EACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/E17-1020.pdf