On Compositional Generalization of Neural Machine Translation

Yafu Li, Yongjing Yin, Yulong Chen, Yue Zhang


Abstract
Modern neural machine translation (NMT) models have achieved competitive performance in standard benchmarks such as WMT. However, there still exist significant issues such as robustness, domain generalization, etc. In this paper, we study NMT models from the perspective of compositional generalization by building a benchmark dataset, CoGnition, consisting of 216k clean and consistent sentence pairs. We quantitatively analyze effects of various factors using compound translation error rate, then demonstrate that the NMT model fails badly on compositional generalization, although it performs remarkably well under traditional metrics.
Anthology ID:
2021.acl-long.368
Volume:
Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
Month:
August
Year:
2021
Address:
Online
Editors:
Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
Venues:
ACL | IJCNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4767–4780
Language:
URL:
https://aclanthology.org/2021.acl-long.368
DOI:
10.18653/v1/2021.acl-long.368
Bibkey:
Cite (ACL):
Yafu Li, Yongjing Yin, Yulong Chen, and Yue Zhang. 2021. On Compositional Generalization of Neural Machine Translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4767–4780, Online. Association for Computational Linguistics.
Cite (Informal):
On Compositional Generalization of Neural Machine Translation (Li et al., ACL-IJCNLP 2021)
Copy Citation:
PDF:
https://aclanthology.org/2021.acl-long.368.pdf
Video:
 https://aclanthology.org/2021.acl-long.368.mp4
Code
 yafuly/CoGnition
Data
CFQROCStoriesSCAN