A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions

Antonio Toral, Víctor M. Sánchez-Cartagena


Abstract
We aim to shed light on the strengths and weaknesses of the newly introduced neural machine translation paradigm. To that end, we conduct a multifaceted evaluation in which we compare outputs produced by state-of-the-art neural machine translation and phrase-based machine translation systems for 9 language directions across a number of dimensions. Specifically, we measure the similarity of the outputs, their fluency and amount of reordering, the effect of sentence length and performance across different error categories. We find out that translations produced by neural machine translation systems are considerably different, more fluent and more accurate in terms of word order compared to those produced by phrase-based systems. Neural machine translation systems are also more accurate at producing inflected forms, but they perform poorly when translating very long sentences.
Anthology ID:
E17-1100
Volume:
Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers
Month:
April
Year:
2017
Address:
Valencia, Spain
Editors:
Mirella Lapata, Phil Blunsom, Alexander Koller
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1063–1073
Language:
URL:
https://aclanthology.org/E17-1100
DOI:
Bibkey:
Cite (ACL):
Antonio Toral and Víctor M. Sánchez-Cartagena. 2017. A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1063–1073, Valencia, Spain. Association for Computational Linguistics.
Cite (Informal):
A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions (Toral & Sánchez-Cartagena, EACL 2017)
Copy Citation:
PDF:
https://aclanthology.org/E17-1100.pdf
Code
 antot/neural_vs_-phrasebased_smt_eacl17
Data
WMT 2016