Unifying Input and Output Smoothing in Neural Machine Translation

Yingbo Gao, Baohao Liao, Hermann Ney


Abstract
Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation systems. Label smoothing is another effective method that penalizes over-confident model outputs by discounting some probability mass from the true target word, smoothing the output of neural machine translation systems. Having the benefit of updating all word vectors in each optimization step and better regularizing the models, the two smoothing methods are shown to bring significant improvements in translation performance. In this work, we study how to best combine the methods and stack the improvements. Specifically, we vary the prior distributions to smooth with, the hyperparameters that control the smoothing strength, and the token selection procedures. We conduct extensive experiments on small datasets, evaluate the recipes on larger datasets, and examine the implications when back-translation is further used. Our results confirm cumulative improvements when input and output smoothing are used in combination, giving up to +1.9 BLEU scores on standard machine translation tasks and reveal reasons why these smoothing methods should be preferred.
Anthology ID:
2020.coling-main.386
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
4361–4372
Language:
URL:
https://aclanthology.org/2020.coling-main.386
DOI:
10.18653/v1/2020.coling-main.386
Bibkey:
Cite (ACL):
Yingbo Gao, Baohao Liao, and Hermann Ney. 2020. Unifying Input and Output Smoothing in Neural Machine Translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4361–4372, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Unifying Input and Output Smoothing in Neural Machine Translation (Gao et al., COLING 2020)
Copy Citation:
PDF:
https://aclanthology.org/2020.coling-main.386.pdf