Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs

Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, Despoina Mouratidis


Abstract
The present study aims to compare three systems: a generic statistical machine translation (SMT), a generic neural machine translation (NMT) and a tailored-NMT system focusing on the English to Greek language pair. The comparison is carried out following a mixed-methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post editing (PE) effort and human error analysis performed by 16 postgraduate Translation students. The findings reveal a higher score for both the generic NMT and the tailored-NMT outputs as regards automatic metrics and human evaluation metrics, with the tailored-NMT output faring even better than the generic NMT output.
Anthology ID:
2020.eamt-1.47
Volume:
Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
Month:
November
Year:
2020
Address:
Lisboa, Portugal
Editors:
André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, Mikel L. Forcada
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation
Note:
Pages:
441–450
Language:
URL:
https://aclanthology.org/2020.eamt-1.47
DOI:
Bibkey:
Cite (ACL):
Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, and Despoina Mouratidis. 2020. Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 441–450, Lisboa, Portugal. European Association for Machine Translation.
Cite (Informal):
Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs (Stasimioti et al., EAMT 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.eamt-1.47.pdf