On Compositional Generalization of Neural Machine Translation

Yafu Li, Yongjing Yin, Yulong Chen, Yue Zhang


Abstract
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.
Anthology ID:
2021.acl-long.368
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4767–4780
Language:
URL:
https://aclanthology.org/2021.acl-long.368
DOI:
10.18653/v1/2021.acl-long.368
Bibkey:
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.368.pdf