Generalizing Back-Translation in Neural Machine Translation

Miguel Graça, Yunsu Kim, Julian Schamper, Shahram Khadivi, Hermann Ney


Abstract
Back-translation — data augmentation by translating target monolingual data — is a crucial component in modern neural machine translation (NMT). In this work, we reformulate back-translation in the scope of cross-entropy optimization of an NMT model, clarifying its underlying mathematical assumptions and approximations beyond its heuristic usage. Our formulation covers broader synthetic data generation schemes, including sampling from a target-to-source NMT model. With this formulation, we point out fundamental problems of the sampling-based approaches and propose to remedy them by (i) disabling label smoothing for the target-to-source model and (ii) sampling from a restricted search space. Our statements are investigated on the WMT 2018 German <-> English news translation task.
Anthology ID:
W19-5205
Volume:
Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)
Month:
August
Year:
2019
Address:
Florence, Italy
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
45–52
Language:
URL:
https://aclanthology.org/W19-5205
DOI:
10.18653/v1/W19-5205
Bibkey:
Cite (ACL):
Miguel Graça, Yunsu Kim, Julian Schamper, Shahram Khadivi, and Hermann Ney. 2019. Generalizing Back-Translation in Neural Machine Translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers), pages 45–52, Florence, Italy. Association for Computational Linguistics.
Cite (Informal):
Generalizing Back-Translation in Neural Machine Translation (Graça et al., WMT 2019)
Copy Citation:
PDF:
https://aclanthology.org/W19-5205.pdf