Manual Analysis of Structurally Informed Reordering in German-English Machine Translation

Teresa Herrmann, Jan Niehues, Alex Waibel


Abstract
Word reordering is a difficult task for translation. Common automatic metrics such as BLEU have problems reflecting improvements in target language word order. However, it is a crucial aspect for humans when deciding on translation quality. This paper presents a detailed analysis of a structure-aware reordering approach applied in a German-to-English phrase-based machine translation system. We compare the translation outputs of two translation systems applying reordering rules based on parts-of-speech and syntax trees on a sentence-by-sentence basis. For each sentence-pair we examine the global translation performance and classify local changes in the translated sentences. This analysis is applied to three data sets representing different genres. While the improvement in BLEU differed substantially between the data sets, the manual evaluation showed that both global translation performance as well as individual types of improvements and degradations exhibit a similar behavior throughout the three data sets. We have observed that for 55-64% of the sentences with different translations, the translation produced using the tree-based reordering was considered to be the better translation. As intended by the investigated reordering model, most improvements are achieved by improving the position of the verb or being able to translate a verb that could not be translated before.
Anthology ID:
L14-1462
Volume:
Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)
Month:
May
Year:
2014
Address:
Reykjavik, Iceland
Editors:
Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association (ELRA)
Note:
Pages:
4379–4386
Language:
URL:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/569_Paper.pdf
DOI:
Bibkey:
Cite (ACL):
Teresa Herrmann, Jan Niehues, and Alex Waibel. 2014. Manual Analysis of Structurally Informed Reordering in German-English Machine Translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 4379–4386, Reykjavik, Iceland. European Language Resources Association (ELRA).
Cite (Informal):
Manual Analysis of Structurally Informed Reordering in German-English Machine Translation (Herrmann et al., LREC 2014)
Copy Citation:
PDF:
http://www.lrec-conf.org/proceedings/lrec2014/pdf/569_Paper.pdf